Skip to content

Commit

Permalink
Adds support for COPY TO/FROM Azure Blob Storage (#55)
Browse files Browse the repository at this point in the history
Supports following Azure Blob uri forms:
- `az://{container}/key`
- `azure://{container}/key`
- `https://{account}.blob.core.windows.net/{container}`

**Configuration**

The simplest way to configure object storage is by creating the standard [`~/.azure/config`](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration?view=azure-cli-latest) file:

```bash
$ cat ~/.azure/config
[storage]
account = devstoreaccount1
key = Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
```

Alternatively, you can use the following environment variables when starting postgres to configure the Azure Blob Storage client:
- `AZURE_STORAGE_ACCOUNT`: the storage account name of the Azure Blob
- `AZURE_STORAGE_KEY`: the storage key of the Azure Blob
- `AZURE_CONNECTION_STRING`: the connection string of the Azure Blob
- `AZURE_STORAGE_SAS_TOKEN`: the storage SAS token for the Azure Blob
- `AZURE_TENANT_ID`: the tenant id for client secret auth **(only via environment variables)**
- `AZURE_CLIENT_ID`: the client id for client secret auth **(only via environment variables)**
- `AZURE_CLIENT_SECRET`: the client secret for client secret auth **(only via environment variables)**
- `AZURE_STORAGE_ENDPOINT`: the endpoint **(only via environment variables)**
- `AZURE_ALLOW_HTTP`: allows http endpoints **(only via environment variables)**
- `AZURE_CONFIG_FILE`: an alternative location for the config file **(only via environment variables)**

**Bonus**
Additionally, PR adds support to the following S3 uri forms:
- `https://s3.amazonaws.com/{bucket}/key`
- `https://{bucket}.s3.amazonaws.com/key`

Closes #50
  • Loading branch information
aykut-bozkurt authored Jan 10, 2025
1 parent 2c1a62d commit fcb5036
Show file tree
Hide file tree
Showing 17 changed files with 1,362 additions and 228 deletions.
10 changes: 10 additions & 0 deletions .devcontainer/.env
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,15 @@ AWS_S3_TEST_BUCKET=testbucket
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin

# Azure Blob tests
AZURE_STORAGE_ACCOUNT=devstoreaccount1
AZURE_STORAGE_KEY="Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;"
AZURE_STORAGE_ENDPOINT=http://localhost:10000/devstoreaccount1
AZURE_ALLOW_HTTP=true
AZURE_TEST_CONTAINER_NAME=testcontainer
AZURE_TEST_READ_ONLY_SAS="se=2100-05-05&sp=r&sv=2022-11-02&sr=c&sig=YMPFnAHKe9y0o3hFegncbwQTXtAyvsJEgPB2Ne1b9CQ%3D"
AZURE_TEST_READ_WRITE_SAS="se=2100-05-05&sp=rcw&sv=2022-11-02&sr=c&sig=TPz2jEz0t9L651t6rTCQr%2BOjmJHkM76tnCGdcyttnlA%3D"

# Others
RUST_TEST_THREADS=1
5 changes: 5 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ RUN apt-get update && apt-get -y install build-essential libreadline-dev zlib1g-
curl lsb-release ca-certificates gnupg sudo git \
nano net-tools awscli

# install azure-cli
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/keyrings/microsoft.gpg > /dev/null
RUN echo "deb [arch=`dpkg --print-architecture` signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ `lsb_release -cs` main" | tee /etc/apt/sources.list.d/azure-cli.list
RUN apt-get update && apt-get install -y azure-cli

# install Postgres
RUN sh -c 'echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
Expand Down
21 changes: 18 additions & 3 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,23 @@ services:
build:
context: .
dockerfile: Dockerfile
command: sleep infinity
entrypoint: "./entrypoint.sh"
network_mode: host
volumes:
- ..:/workspace
- ${USERPROFILE}${HOME}/.ssh:/home/rust/.ssh:ro
- ${USERPROFILE}${HOME}/.ssh/known_hosts:/home/rust/.ssh/known_hosts:rw
- ${USERPROFILE}${HOME}/.gitconfig:/home/rust/.gitconfig:ro
- ${USERPROFILE}${HOME}/.aws:/home/rust/.aws:ro
- ${USERPROFILE}${HOME}/.aws:/home/rust/.aws:rw
- ${USERPROFILE}${HOME}/.azure:/home/rust/.azure:rw
- ./entrypoint.sh:/entrypoint.sh
env_file:
- .env
cap_add:
- SYS_PTRACE
depends_on:
- minio
- azurite

minio:
image: minio/minio
Expand All @@ -31,4 +34,16 @@ services:
timeout: 2s
retries: 3
volumes:
- ./minio-entrypoint.sh:/entrypoint.sh
- ./minio-entrypoint.sh:/entrypoint.sh

azurite:
image: mcr.microsoft.com/azure-storage/azurite
env_file:
- .env
network_mode: host
restart: unless-stopped
healthcheck:
test: ["CMD", "nc", "-z", "localhost", "10000"]
interval: 6s
timeout: 2s
retries: 3
8 changes: 8 additions & 0 deletions .devcontainer/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#!/bin/bash

trap "echo 'Caught termination signal. Exiting...'; exit 0" SIGINT SIGTERM

# create azurite container
az storage container create -n $AZURE_TEST_CONTAINER_NAME --connection-string $AZURE_STORAGE_CONNECTION_STRING

sleep infinity
21 changes: 20 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@ jobs:
postgresql-client-${{ env.PG_MAJOR }} \
libpq-dev
- name: Install azure-cli
run: |
curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/keyrings/microsoft.gpg > /dev/null
echo "deb [arch=`dpkg --print-architecture` signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ `lsb_release -cs` main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
sudo apt-get update && sudo apt-get install -y azure-cli
- name: Install and configure pgrx
run: |
Expand Down Expand Up @@ -112,14 +117,28 @@ jobs:
-p 9000:9000 \
--entrypoint "./entrypoint.sh" \
--volume ./.devcontainer/minio-entrypoint.sh:/entrypoint.sh \
--name miniocontainer \
minio/minio
while ! curl $AWS_ENDPOINT_URL; do
echo "Waiting for $AWS_ENDPOINT_URL..."
sleep 1
done
- name: Start Azurite for Azure Blob Storage emulator tests
run: |
docker run -d \
--env-file .devcontainer/.env \
-p 10000:10000 \
mcr.microsoft.com/azure-storage/azurite
while ! curl $AZURE_STORAGE_ENDPOINT; do
echo "Waiting for $AZURE_STORAGE_ENDPOINT..."
sleep 1
done
# create container
az storage container create -n $AZURE_TEST_CONTAINER_NAME --connection-string $AZURE_STORAGE_CONNECTION_STRING
- name: Run tests
run: |
# Run tests with coverage tool
Expand Down
Loading

0 comments on commit fcb5036

Please sign in to comment.