needlestack.servicers package¶

Submodules¶

needlestack.servicers.factory module¶

needlestack.servicers.factory.create_server(config)[source]¶

Create a gRPC server app with a health servicer.

Parameters: config (BaseConfig) – Config with settings on how to setup the server
Return type: _Server

needlestack.servicers.factory.create_zookeeper_cluster_manager(config)[source]¶

Create a Zookeeper client for cluster managment.

Parameters: config (BaseConfig) – Config with settings on how to set up a Zookeeper client
Return type: ClusterManager

needlestack.servicers.factory.serve(server)[source]¶

needlestack.servicers.logging module¶

needlestack.servicers.logging.configure_logger(config)[source]¶

Configure the base logger using a configuration class. Sets the log level, debug handler, and file handler.

Parameters: config (BaseConfig) – Config class that defines how to log

needlestack.servicers.logging.get_debug_handler(fmt, datefmt)[source]¶

Get a debug stdout logging handler

Parameters

fmt (str) – Logging format string
datefmt (str) – Date format

needlestack.servicers.logging.get_file_handler(fmt, datefmt, log_file, max_bytes, backup_count)[source]¶

Get a rotating file logging handler

Parameters

fmt (str) – Logging format string
datefmt (str) – Date format
log_file (str) – Path to log file
max_bytes (int) – Number of builts per log file
backup_count (int) – Number of log files to keep in rotation

needlestack.servicers.merger module¶

class needlestack.servicers.merger.MergerServicer(config, cluster_manager)[source]¶

Bases: needlestack.apis.servicers_pb2_grpc.MergerServicer

A gRPC servicer to accept external requests, use searcher nodes, and merge results together.

CollectionsAdd(request, context)[source]¶

CollectionsDelete(request, context)[source]¶

CollectionsList(request, context)[source]¶

CollectionsLoad(request, context)[source]¶

Retrieve(request, context)[source]¶

Search(request, context)[source]¶

collections_load()[source]¶

Return type: bool

get_searcher_hostports(collection_name, shard_names=None)[source]¶

Return type: List[Tuple[str, List[str]]]

get_searcher_stub(hostport)[source]¶

Return type: SearcherStub

needlestack.servicers.searcher module¶

class needlestack.servicers.searcher.SearcherServicer(config, cluster_manager)[source]¶

Bases: needlestack.apis.servicers_pb2_grpc.SearcherServicer

A gRPC servicer to perform kNN queries on in-memory index structures

CollectionsLoad(request, context)[source]¶

Retrieve(request, context)[source]¶

Search(request, context)[source]¶

get_collection(name)[source]¶

Return type: Collection

load_collections()[source]¶

Load collections from Zookeeper configs

There are 4 states to handle for each collection:

A new collection needs to be loaded
An existing collection needs to be dropped
An existing collection added/dropped shards
No changes

needlestack.servicers.settings module¶

class needlestack.servicers.settings.BaseConfig[source]¶

Bases: object

Base configuration for gRPC services

DEBUG¶: Attach a stream handler to console for logger

DEBUG_LOG_FORMAT¶: Format string for debug logger

LOG_LEVEL¶: Level for logger

LOG_FORMAT_DATE¶: Format string for date

LOG_FILE¶: Filepath to log file

LOG_FILE_BACKUPS¶: Number of log files to keep in rotation

LOG_FILE_LOG_FORMAT¶: Format string for file logger

LOG_FILE_MAX_BYTES¶: Max byte size for log file

MAX_WORKERS¶: Number of worker threads per gRPC server

HOSTNAME¶: Hostname of node

SERVICER_PORT¶: Port of gRPC server

MUTUAL_TLS¶: Require server and client to authenticate each other the CA

SSL_CA_CERT_CHAIN¶: Certificate authority certificate chain bytes

SSL_CA_CERT_CHAIN_FILE¶: Certificate authority certificate chain file

SSL_SERVER_PRIVATE_KEY¶: Server private key bytes

SSL_SERVER_PRIVATE_KEY_FILE¶: Server private key file

SSL_SERVER_CERT_CHAIN¶: Server certificate chain bytes

SSL_SERVER_CERT_CHAIN_FILE¶: Server certificate chain file

SSL_CLIENT_PRIVATE_KEY¶: Client private key bytes

SSL_CLIENT_PRIVATE_KEY_FILE¶: Client private key file

SSL_CLIENT_CERT_CHAIN¶: Client certificate chain bytes

SSL_CLIENT_CERT_CHAIN_FILE¶: Client certificate chain file

CLUSTER_NAME¶: Name for Needlestack cluster

ZOOKEEPER_ROOT¶: Root path on Zookeeper

ZOOKEEPER_HOSTS¶: List of Zookeeper host for cluster manager

hostport¶: Hostport to gRPC server

use_mutual_tls¶: Should server and clients be authenticated

use_server_ssl¶: Should server be authenticated

ssl_server_credentials¶: gRPC SSL server credentials

DEBUG = False

DEBUG_LOG_FORMAT = '%(asctime)s [%(name)s] [%(threadName)-10s] [%(levelname)s] - %(message)s'

LOG_FILE = None

LOG_FILE_LOG_FORMAT = '%(asctime)s [%(name)s] [%(thread)d] [%(process)d] [%(levelname)s] - %(message)s'

LOG_FORMAT_DATE = '%Y-%m-%d %H:%M:%S'

LOG_LEVEL = 'WARNING'

MUTUAL_TLS = False

SSL_CA_CERT_CHAIN = None

SSL_CA_CERT_CHAIN_FILE = None

SSL_CLIENT_CERT_CHAIN = None

SSL_CLIENT_CERT_CHAIN_FILE = None

SSL_CLIENT_PRIVATE_KEY = None

SSL_CLIENT_PRIVATE_KEY_FILE = None

SSL_SERVER_CERT_CHAIN = None

SSL_SERVER_CERT_CHAIN_FILE = None

SSL_SERVER_PRIVATE_KEY = None

SSL_SERVER_PRIVATE_KEY_FILE = None

ZOOKEEPER_ROOT = '/needlestack'

property ca_certificate¶

Return type: Optional[bytes]

property channel_certificate¶

Return type: Optional[bytes]

property channel_private_key¶

Return type: Optional[bytes]

property hostport

Return type: str

property server_certificate¶

Return type: Optional[bytes]

property server_private_key¶

Return type: Optional[bytes]

property ssl_channel_credentials¶

Return type: Optional[ChannelCredentials]

property ssl_server_credentials

Return type: Optional[ServerCredentials]

property use_channel_ssl¶

property use_mutual_tls

Return type: bool

property use_server_ssl

Return type: bool

class needlestack.servicers.settings.TestConfig[source]¶

Bases: needlestack.servicers.settings.BaseConfig

Configs for local test environment

CLUSTER_NAME = 'test_needlestack'¶

DEBUG = True¶

HOSTNAME = 'localhost'¶

LOG_FILE = '/tmp/needlestack.log'¶

LOG_FILE_BACKUPS = 2¶

LOG_FILE_MAX_BYTES = 1048576¶

LOG_LEVEL = 'DEBUG'¶

MAX_WORKERS = 2¶

SERVICER_PORT = 50051¶

ZOOKEEPER_HOSTS = ['zoo1:2181', 'zoo2:2181', 'zoo3:2181']¶