needlestack.data_sources package

Submodules

needlestack.data_sources.gcs module

class needlestack.data_sources.gcs.GcsDataSource

Bases: needlestack.data_sources.source.DataSource

Data source that lives locally on disk as a file.

bucket_name

Google Cloud Storage bucket name

blob_name

Blob name in bucket

project_name

Google Cloud Platform project name

credentials_file

JSON credentials file for GCP. If not provided, the google.cloud package will try to get the credentials implicitly

property blob

Return type

Blob

get_content(mode='rb')

Yield raw data from the data source

property last_modified

Last time a data source was modified

local_filename()

Yield a local filename to access the data as file

populate_from_proto(proto)

Populate DataSource from protobuf defining the data source

Parameters

proto – Protobuf to populate fields for DataSource implementation

needlestack.data_sources.gcs.get_client(credentials_file=None)

Gets a cached GCS client, or creates and caches a GCS client. If not provided a credentials file, checks the environment variables for client authentication.

Parameters

credentials_file (Optional[str]) – Path to GCP credentials json

Return type

Client

needlestack.data_sources.local module

class needlestack.data_sources.local.LocalDataSource

Bases: needlestack.data_sources.source.DataSource

Data source that lives locally on disk as a file

filename

Filename on disk

get_content(mode='rb')

Yield raw data from the data source

property last_modified

Last time a data source was modified

local_filename()

Yield a local filename to access the data as file

populate_from_proto(proto)

Populate DataSource from protobuf defining the data source

Parameters

proto – Protobuf to populate fields for DataSource implementation

needlestack.data_sources.source module

class needlestack.data_sources.source.DataSource

Bases: object

Base class for data source implementations. Defines interfaces for populating access data.

static from_proto(proto)

Factory method to construct the correct implementation of a DataSource from a protobuf.

Parameters

proto (DataSource) – Protobuf defining how to access data

Return type

DataSource

get_content(mode)

Yield raw data from the data source

property last_modified

Last time a data source was modified

Return type

float

local_filename()

Yield a local filename to access the data as file

populate_from_proto(proto)

Populate DataSource from protobuf defining the data source

Parameters

proto (DataSource) – Protobuf to populate fields for DataSource implementation

Module contents