GRPC API

• consumer_service.proto

  • ConsumerApi

  • Consumer
  • CreateConsumerRequest
  • CreateConsumerResponse
  • DeleteConsumerRequest
  • DeleteConsumerResponse
  • GetConsumerRequest
  • GetConsumerResponse
  • GetConsumersRequest
  • GetConsumersResponse
• dag_pb.proto
  • PBLink
  • PBNode
• index_service.proto

  • IndexApi

  • AttachFileEngineRequest
  • AttachIndexRequest
  • AttachIndexResponse
  • AttachRemoteEngineRequest
  • CacheConfig
  • CommitIndexRequest
  • CommitIndexResponse
  • CopyDocumentsRequest
  • CopyDocumentsResponse
  • CopyIndexRequest
  • CopyIndexResponse
  • CreateFileEngineRequest
  • CreateIndexRequest
  • CreateIndexResponse
  • CreateMemoryEngineRequest
  • DeleteDocumentsRequest
  • DeleteDocumentsResponse
  • DeleteIndexRequest
  • DeleteIndexResponse
  • DocumentsRequest
  • DocumentsResponse
  • FileEngineConfig
  • GetIndexRequest
  • GetIndexResponse
  • GetIndicesAliasesRequest
  • GetIndicesAliasesResponse
  • GetIndicesAliasesResponse.IndicesAliasesEntry
  • GetIndicesRequest
  • GetIndicesResponse
  • IndexAttributes
  • IndexDescription
  • IndexDocumentOperation
  • IndexDocumentRequest
  • IndexDocumentResponse
  • IndexDocumentStreamRequest
  • IndexDocumentStreamResponse
  • IndexEngineConfig
  • IndexOperation
  • LogMergePolicy
  • MappedField
  • MemoryEngineConfig
  • MergePolicy
  • MergeSegmentsRequest
  • MergeSegmentsResponse
  • RemoteEngineConfig
  • RemoteEngineConfig.HeadersTemplateEntry
  • SetIndexAliasRequest
  • SetIndexAliasResponse
  • SortByField
  • TemporalMergePolicy
  • VacuumIndexRequest
  • VacuumIndexResponse
  • WarmupIndexRequest

  • WarmupIndexResponse

  • Compression
  • ConflictStrategy
• public_service.proto
  • PublicApi
• query.proto
  • AggregationCollector
  • AggregationCollectorOutput
  • AllQuery
  • BooleanQuery
  • BooleanSubquery
  • BoostQuery
  • Collector
  • CollectorOutput
  • CountCollector
  • CountCollectorOutput
  • DisjunctionMaxQuery
  • DocumentsCollectorOutput
  • EmptyQuery
  • ExactMatchesPromoter
  • ExistsQuery
  • FacetCollector
  • FacetCollectorOutput
  • FacetCollectorOutput.FacetCountsEntry
  • Highlight
  • MatchQuery
  • MatchQueryBooleanShouldMode
  • MatchQueryDisjuctionMaxMode
  • MoreLikeThisQuery
  • MorphologyConfig
  • NerMatchesPromoter
  • PhraseQuery
  • Query
  • QueryParserConfig
  • QueryParserConfig.FieldAliasesEntry
  • QueryParserConfig.FieldBoostsEntry
  • QueryParserConfig.MorphologyConfigsEntry
  • QueryParserConfig.TermFieldMapperConfigsEntry
  • RandomDocument
  • Range
  • RangeQuery
  • RegexQuery
  • ReservoirSamplingCollector
  • ReservoirSamplingCollectorOutput
  • Score
  • ScoredDocument
  • ScoredDocument.SnippetsEntry
  • Scorer
  • SearchRequest
  • SearchResponse
  • Snippet
  • TermFieldMapperConfig
  • TermQuery
  • TopDocsCollector

  • TopDocsCollector.SnippetConfigsEntry

  • Occur
• reflection_service.proto

  • ReflectionApi

  • GetTopTermsRequest
  • GetTopTermsResponse
  • GetTopTermsResponse.PerSegmentEntry
  • SegmentTerms
  • TermInfo
• search_service.proto
  • SearchApi
• unixfs.proto
  • Data

  • Metadata

  • Data.DataType
• utils.proto

  • Empty

  • Order

consumer_service.proto

Consumer

Consumer description

Field	Type	Label	Description
consumer_name	string		Consumer name
index_name	string		Summa index_name

CreateConsumerRequest

Request describe how new Consumer should be created

Field	Type	Label	Description
bootstrap_servers	string	repeated	Kafka boostrap servers
group_id	string		Kafka group ID
index_name	string		Summa index_name which will ingest data from Kafka topics
consumer_name	string		Consumer name, used for further referencing consumer in API and configs
topics	string	repeated	List of topics to consume

CreateConsumerResponse

Field	Type	Label	Description
consumer	Consumer

DeleteConsumerRequest

Field	Type	Label	Description
consumer_name	string

DeleteConsumerResponse

Field	Type	Label	Description
consumer_name	string

GetConsumerRequest

Field	Type	Label	Description
index_name	string
consumer_name	string

GetConsumerResponse

Field	Type	Label	Description
consumer	Consumer

GetConsumersRequest

GetConsumersResponse

Field	Type	Label	Description
consumers	Consumer	repeated

ConsumerApi

Manage ingestion data from Kafka

Method Name	Request Type	Response Type	Description
create_consumer	CreateConsumerRequest	CreateConsumerResponse	Create a new consumer
get_consumer	GetConsumerRequest	GetConsumerResponse	Get a single consumer
get_consumers	GetConsumersRequest	GetConsumersResponse	Get a list of all consumers
delete_consumer	DeleteConsumerRequest	DeleteConsumerResponse	Remove a consumer

dag_pb.proto

PBLink

Field	Type	Label	Description
hash	bytes	optional	binary CID (with no multibase prefix) of the target object
name	string	optional	UTF-8 string name
t_size	uint64	optional	cumulative size of target object

PBNode

Field	Type	Label	Description
links	PBLink	repeated	refs to other objects
data	bytes	optional	opaque user data

index_service.proto

AttachFileEngineRequest

Attach file engine request

AttachIndexRequest

Attach index request

Field	Type	Label	Description
index_name	string		Index name for attaching
file	AttachFileEngineRequest
remote	AttachRemoteEngineRequest
merge_policy	MergePolicy
query_parser_config	QueryParserConfig

AttachIndexResponse

Description of the attached index

Field	Type	Label	Description
index	IndexDescription

AttachRemoteEngineRequest

Attach remote engine request

Field	Type	Label	Description
config	RemoteEngineConfig

CacheConfig

Field	Type	Label	Description
cache_size	uint64		Total cache size in bytes

CommitIndexRequest

Store the state of index to the storage

Field	Type	Label	Description
index_name	string
with_hotcache	bool

CommitIndexResponse

Returned data from the commit command

Field	Type	Label	Description
elapsed_secs	double		Pure time spent for committing

CopyDocumentsRequest

Copy documents from one index to another. Their schemes must be compatible

Field	Type	Label	Description
source_index_name	string		Where documents should be taken from
target_index_name	string		Where documents should be copied to
conflict_strategy	ConflictStrategy	optional	How to deal with conflicts on unique fields. Recommended to set to DoNothing for large updates and maintain uniqueness in your application

CopyDocumentsResponse

Copy documents response

Field	Type	Label	Description
elapsed_secs	double
copied_documents	uint32

CopyIndexRequest

Request that changes index engine. Currently possible to convert File to IPFS

Field	Type	Label	Description
source_index_name	string		Name of index that will be migrated. It will be left intact after migration.
target_index_name	string		Name of index that will be created
file	CreateFileEngineRequest
memory	CreateMemoryEngineRequest
merge_policy	MergePolicy

CopyIndexResponse

Response describing migrated index

Field	Type	Label	Description
index	IndexDescription

CreateFileEngineRequest

CreateIndexRequest

Request for index creation

Field	Type	Label	Description
index_name	string		Index name
file	CreateFileEngineRequest
memory	CreateMemoryEngineRequest
schema	string		Index schema in Tantivy format
compression	Compression		Compression for store
blocksize	uint32	optional	Size of store blocks
index_attributes	IndexAttributes		Optional index fields
merge_policy	MergePolicy		Merge policy
query_parser_config	QueryParserConfig

CreateIndexResponse

Field	Type	Label	Description
index	IndexDescription

CreateMemoryEngineRequest

DeleteDocumentsRequest

Field	Type	Label	Description
index_name	string
query	Query

DeleteDocumentsResponse

Field	Type	Label	Description
deleted_documents	uint64

DeleteIndexRequest

Field	Type	Label	Description
index_name	string

DeleteIndexResponse

Field	Type	Label	Description
deleted_index_name	string

DocumentsRequest

Request a stream of all documents from the index

Field	Type	Label	Description
index_name	string
fields	string	repeated
query_filter	Query	optional

DocumentsResponse

Single document from the index

Field	Type	Label	Description
document	string

FileEngineConfig

Field	Type	Label	Description
path	string

GetIndexRequest

Field	Type	Label	Description
index_name	string

GetIndexResponse

Field	Type	Label	Description
index	IndexDescription

GetIndicesAliasesRequest

GetIndicesAliasesResponse

Field	Type	Label	Description
indices_aliases	GetIndicesAliasesResponse.IndicesAliasesEntry	repeated

GetIndicesAliasesResponse.IndicesAliasesEntry

Field	Type	Label	Description
key	string
value	string

GetIndicesRequest

GetIndicesResponse

Field	Type	Label	Description
index_names	string	repeated

IndexAttributes

Field	Type	Label	Description
created_at	uint64		Timestamp when index has been created
unique_fields	string	repeated	Unique fields of the index. Summa maintains unique constraint on them and uses for deduplicating data
multi_fields	string	repeated	Multi fields is ones that may have multiple values and processed as lists. All other fields will be forcefully converted to singular value
description	string	optional	Text index description
conflict_strategy	ConflictStrategy
mapped_fields	MappedField	repeated

IndexDescription

Description containing Index metadata fields

Field	Type	Label	Description
index_name	string
index_aliases	string	repeated	All index aliases
index_engine	IndexEngineConfig
num_docs	uint64		The number of committed documents
compression	Compression		Used compression for store
index_attributes	IndexAttributes		All custom index attributes

IndexDocumentOperation

Indexing operations that contains document serialized in JSON format

Field	Type	Label	Description
document	bytes

IndexDocumentRequest

Field	Type	Label	Description
index_name	string
document	bytes

IndexDocumentResponse

IndexDocumentStreamRequest

Field	Type	Label	Description
index_name	string
documents	bytes	repeated
conflict_strategy	ConflictStrategy	optional

IndexDocumentStreamResponse

Field	Type	Label	Description
elapsed_secs	double
success_docs	uint64
failed_docs	uint64

IndexEngineConfig

Description of the IndexEngine responsible for managing files in the persistent storage

Field	Type	Label	Description
file	FileEngineConfig
memory	MemoryEngineConfig
remote	RemoteEngineConfig
merge_policy	MergePolicy		Merge policy
query_parser_config	QueryParserConfig

IndexOperation

Message that should be put in Kafka for ingesting by Summa consumers

Field	Type	Label	Description
index_document	IndexDocumentOperation

LogMergePolicy

Merge policy for implementing LogMergePolicy

Field	Type	Label	Description
is_frozen	bool		Set if once merged segment should be left intact

MappedField

Field	Type	Label	Description
source_field	string
target_field	string

MemoryEngineConfig

Field	Type	Label	Description
schema	string		Schema of the index for memory engine

MergePolicy

Merge policy that describes how to merge committed segments

Field	Type	Label	Description
log	LogMergePolicy
temporal	TemporalMergePolicy

MergeSegmentsRequest

Field	Type	Label	Description
index_name	string
segment_ids	string	repeated

MergeSegmentsResponse

Field	Type	Label	Description
segment_id	string	optional

RemoteEngineConfig

Remote HTTP engine config

Field	Type	Label	Description
method	string		Which method should be used to request remote endpoint
url_template	string		URL template which will be used to generate real URL by variables substitution
headers_template	RemoteEngineConfig.HeadersTemplateEntry	repeated	Headers template which will be used to generate real URL by variables substitution
cache_config	CacheConfig		Description of the cache for the engine
timeout_ms	uint32	optional	Timeout for the request

RemoteEngineConfig.HeadersTemplateEntry

Field	Type	Label	Description
key	string
value	string

SetIndexAliasRequest

Field	Type	Label	Description
index_alias	string
index_name	string

SetIndexAliasResponse

Field	Type	Label	Description
old_index_name	string	optional	If set, equals to the previous alias of the index

SortByField

Field	Type	Label	Description
field	string
order	Order

TemporalMergePolicy

Merge policy for compressing old segments

Field	Type	Label	Description
merge_older_then_secs	uint64

VacuumIndexRequest

Field	Type	Label	Description
index_name	string
excluded_segments	string	repeated

VacuumIndexResponse

Field	Type	Label	Description
freed_space_bytes	uint64

WarmupIndexRequest

Field	Type	Label	Description
index_name	string
is_full	bool		If set to false, only term dictionaries will be warmed, otherwise the entire index will be read.

WarmupIndexResponse

Field	Type	Label	Description
elapsed_secs	double		Time spent in warming operation

Compression

Compression library for store, implies on both performance and occupied disk space

Name	Number	Description
None	0
Zstd	4
Zstd7	5
Zstd9	6
Zstd14	7
Zstd19	8
Zstd22	9

ConflictStrategy

Name	Number	Description
DO_NOTHING	0
OVERWRITE_ALWAYS	1
OVERWRITE	2
MERGE	3

IndexApi

Manages indices

Method Name	Request Type	Response Type	Description
attach_index	AttachIndexRequest	AttachIndexResponse	Attaches index to Summa server. Attaching allows to incorporate and start using of downloaded or network indices
commit_index	CommitIndexRequest	CommitIndexResponse	Committing all collected writes to the index
copy_documents	CopyDocumentsRequest	CopyDocumentsResponse	Copy documents from one index to another
create_index	CreateIndexRequest	CreateIndexResponse	Creates new index from scratch
copy_index	CopyIndexRequest	CopyIndexResponse	Creates new index from scratch
delete_documents	DeleteDocumentsRequest	DeleteDocumentsResponse	Deletes single document from the index by its primary key (therefore, index must have primary key)
delete_index	DeleteIndexRequest	DeleteIndexResponse	Deletes index and physically removes file in the case of FileEngine
documents	DocumentsRequest	DocumentsResponse stream	Stream of all documents from the index
get_indices_aliases	GetIndicesAliasesRequest	GetIndicesAliasesResponse	Gets all existing index aliases
get_index	GetIndexRequest	GetIndexResponse	Gets index description
get_indices	GetIndicesRequest	GetIndicesResponse	Gets all existing index descriptions
index_document_stream	IndexDocumentStreamRequest stream	IndexDocumentStreamResponse	Adds document to the index in a streaming way
index_document	IndexDocumentRequest	IndexDocumentResponse	Adds document to the index
merge_segments	MergeSegmentsRequest	MergeSegmentsResponse	Merges multiple segments into a single one. Used for service purposes
set_index_alias	SetIndexAliasRequest	SetIndexAliasResponse	Sets or replaces existing index alias
vacuum_index	VacuumIndexRequest	VacuumIndexResponse	Removes deletions from all segments
warmup_index	WarmupIndexRequest	WarmupIndexResponse	Loads all hot parts of the index into the memory

public_service.proto

PublicApi

Searches documents in the stored indices

Method Name	Request Type	Response Type	Description
search	SearchRequest	SearchResponse	Make search in Summa

query.proto

AggregationCollector

Field	Type	Label	Description
aggregations	string

AggregationCollectorOutput

Field	Type	Label	Description
aggregation_results	string

AllQuery

BooleanQuery

Field	Type	Label	Description
subqueries	BooleanSubquery	repeated

BooleanSubquery

Field	Type	Label	Description
occur	Occur
query	Query

BoostQuery

Field	Type	Label	Description
query	Query
score	string

Collector

Collectors and CollectorOutputs

Field	Type	Label	Description
top_docs	TopDocsCollector
reservoir_sampling	ReservoirSamplingCollector
count	CountCollector
facet	FacetCollector
aggregation	AggregationCollector

CollectorOutput

Field	Type	Label	Description
documents	DocumentsCollectorOutput
count	CountCollectorOutput
facet	FacetCollectorOutput
aggregation	AggregationCollectorOutput

CountCollector

CountCollectorOutput

Field	Type	Label	Description
count	uint32

DisjunctionMaxQuery

Field	Type	Label	Description
disjuncts	Query	repeated
tie_breaker	string

DocumentsCollectorOutput

Field	Type	Label	Description
scored_documents	ScoredDocument	repeated
has_next	bool

EmptyQuery

ExactMatchesPromoter

Field	Type	Label	Description
slop	uint32
boost	float	optional
fields	string	repeated

ExistsQuery

Field	Type	Label	Description
field	string

FacetCollector

Field	Type	Label	Description
field	string
facets	string	repeated

FacetCollectorOutput

Field	Type	Label	Description
facet_counts	FacetCollectorOutput.FacetCountsEntry	repeated

FacetCollectorOutput.FacetCountsEntry

Field	Type	Label	Description
key	string
value	uint64

Highlight

Field	Type	Label	Description
from	uint32
to	uint32

MatchQuery

Field	Type	Label	Description
value	string
query_parser_config	QueryParserConfig	optional

MatchQueryBooleanShouldMode

MatchQueryDisjuctionMaxMode

Field	Type	Label	Description
tie_breaker	float

MoreLikeThisQuery

Field	Type	Label	Description
document	string
min_doc_frequency	uint64	optional
max_doc_frequency	uint64	optional
min_term_frequency	uint64	optional
max_query_terms	uint64	optional
min_word_length	uint64	optional
max_word_length	uint64	optional
boost	string	optional
stop_words	string	repeated

MorphologyConfig

Field	Type	Label	Description
derive_tenses_coefficient	float	optional

NerMatchesPromoter

Field	Type	Label	Description
boost	float	optional
fields	string	repeated

PhraseQuery

Field	Type	Label	Description
field	string
value	string
slop	uint32

Query

Recursive query DSL

Field	Type	Label	Description
boolean	BooleanQuery
match	MatchQuery
regex	RegexQuery
term	TermQuery
phrase	PhraseQuery
range	RangeQuery
all	AllQuery
more_like_this	MoreLikeThisQuery
boost	BoostQuery
disjunction_max	DisjunctionMaxQuery
empty	EmptyQuery
exists	ExistsQuery

QueryParserConfig

Field	Type	Label	Description
field_aliases	QueryParserConfig.FieldAliasesEntry	repeated
field_boosts	QueryParserConfig.FieldBoostsEntry	repeated
term_field_mapper_configs	QueryParserConfig.TermFieldMapperConfigsEntry	repeated
term_limit	uint32
default_fields	string	repeated
boolean_should_mode	MatchQueryBooleanShouldMode
disjuction_max_mode	MatchQueryDisjuctionMaxMode
exact_matches_promoter	ExactMatchesPromoter
excluded_fields	string	repeated
morphology_configs	QueryParserConfig.MorphologyConfigsEntry	repeated
query_language	string	optional

QueryParserConfig.FieldAliasesEntry

Field	Type	Label	Description
key	string
value	string

QueryParserConfig.FieldBoostsEntry

Field	Type	Label	Description
key	string
value	float

QueryParserConfig.MorphologyConfigsEntry

Field	Type	Label	Description
key	string
value	MorphologyConfig

QueryParserConfig.TermFieldMapperConfigsEntry

Field	Type	Label	Description
key	string
value	TermFieldMapperConfig

RandomDocument

Field	Type	Label	Description
document	string
score	Score
index_alias	string

Range

Field	Type	Label	Description
left	string
right	string
including_left	bool
including_right	bool

RangeQuery

Field	Type	Label	Description
field	string
value	Range

RegexQuery

Field	Type	Label	Description
field	string
value	string

ReservoirSamplingCollector

Field	Type	Label	Description
limit	uint32
fields	string	repeated
excluded_fields	string	repeated

ReservoirSamplingCollectorOutput

Field	Type	Label	Description
documents	RandomDocument	repeated

Score

Field	Type	Label	Description
f64_score	double
u64_score	uint64

ScoredDocument

Field	Type	Label	Description
document	string
score	Score
position	uint32
snippets	ScoredDocument.SnippetsEntry	repeated
index_alias	string

ScoredDocument.SnippetsEntry

Field	Type	Label	Description
key	string
value	Snippet

Scorer

Field	Type	Label	Description
eval_expr	string
order_by	string

SearchRequest

Field	Type	Label	Description
index_alias	string		The index name or alias
query	Query		Query DSL. Use MatchQuery to pass a free-form query
collectors	Collector	repeated	Every collector is responsible of processing and storing documents and/or their derivatives (like counters) to return them to the caller
is_fieldnorms_scoring_enabled	bool	optional	Is requiring fieldnorms needed for the query?
load_cache	bool	optional
store_cache	bool	optional

SearchResponse

Field	Type	Label	Description
elapsed_secs	double		Time spent inside of search handler
collector_outputs	CollectorOutput	repeated	An array of collector outputs

Snippet

Field	Type	Label	Description
fragment	bytes
highlights	Highlight	repeated
html	string

TermFieldMapperConfig

Field	Type	Label	Description
fields	string	repeated

TermQuery

Field	Type	Label	Description
field	string
value	string

TopDocsCollector

Field	Type	Label	Description
limit	uint32
offset	uint32
scorer	Scorer	optional
snippet_configs	TopDocsCollector.SnippetConfigsEntry	repeated
explain	bool
fields	string	repeated
excluded_fields	string	repeated

TopDocsCollector.SnippetConfigsEntry

Field	Type	Label	Description
key	string
value	uint32

Occur

Name	Number	Description
should	0
must	1
must_not	2

reflection_service.proto

GetTopTermsRequest

Field	Type	Label	Description
index_name	string
field_name	string
top_k	uint32

GetTopTermsResponse

Field	Type	Label	Description
per_segment	GetTopTermsResponse.PerSegmentEntry	repeated

GetTopTermsResponse.PerSegmentEntry

Field	Type	Label	Description
key	string
value	SegmentTerms

SegmentTerms

Field	Type	Label	Description
term_infos	TermInfo	repeated

TermInfo

Field	Type	Label	Description
key	bytes
doc_freq	uint32

ReflectionApi

Analyzes indices

Method Name	Request Type	Response Type	Description
get_top_terms	GetTopTermsRequest	GetTopTermsResponse

search_service.proto

SearchApi

Searches documents in the stored indices

Method Name	Request Type	Response Type	Description
search	SearchRequest	SearchResponse	Make search in Summa

unixfs.proto

Data

Field	Type	Label	Description
type	Data.DataType
data	bytes	optional
filesize	uint64	optional
blocksizes	uint64	repeated
hashType	uint64	optional
fanout	uint64	optional

Metadata

Field	Type	Label	Description
MimeType	string	optional

Data.DataType

Name	Number	Description
Raw	0
Directory	1
File	2
Metadata	3
Symlink	4
HAMTShard	5

utils.proto

Empty

Order

Name	Number	Description
Asc	0
Desc	1