Getting started (#1)

* First commit

* Add languages and tests, remove black

Add languages, example usage, and tests. Backend is not actually in yet but its much better than nothing. Black and Ruff were arguing about formatting and I prefer ruff's style

* Format and add dependencies

* Format

* Start adding docs

* Get API technically working

I just need to uncomment a line and update the tests

* Update tests

* Add lang from so example

* Update tests and format

* Remove unused code

* Continue work on docs

* Finish docs

.editorconfig

@@ -0,0 +1,11 @@
+# top-most EditorConfig file
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+
+[*.py]
+charset = utf-8
+indent_style = space
+indent_size = 4

.github/ISSUE_TEMPLATE/bug-report---feature-idea.md

@@ -0,0 +1,38 @@
+---
+name: Bug Report | Feature Idea
+about: Report a bug or suggest a feature
+title: "Bug Report | Feature Idea:"
+labels: bug, enhancement
+assignees: ""
+---
+
+**Bug Report:**
+
+Bug explanation: [Insert here]
+How to reproduce: [Insert here (if not applicable use "N/A")]
+What should happen instead: [Insert here (if not applicable use "N/A")]
+Minimum reproducible example:
+
+```python
+[Insert here (if not applicable use "N/A")]
+```
+
+Traceback:
+
+```
+(if not applicable use "N/A")
+```
+
+<details>
+
+OS: [Insert here (if not applicable use "N/A")]
+Git branch: [Insert here (if not applicable use "N/A")]
+Commit that bug started: [Insert here (if not applicable use "N/A")]
+
+</details>
+
+---
+
+Feature report:
+
+Explain your idea, how it could be implemented, and why it should be added. The more detail you give, the more likely it is to be added. Code snippets that could help add it are _highly recommend_.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,4 @@
+Fixes #__.
+
+Changes in PR:
+ - ...

.github/workflows/isort.yml

@@ -0,0 +1,17 @@
+name: Isort Formatting
+
+on: push
+
+jobs:
+  formatting-isort:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+
+      - name: Code Formatting (App)
+        run: |
+          pip install -r requirements-dev.txt
+          isort --check .

.github/workflows/pypi.yml

@@ -0,0 +1,29 @@
+name: Upload salve to Pypi
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+      - name: Build and publish to Pypi
+        env:
+          TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+          TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+        run: |
+          python -m build --sdist
+          python -m build --wheel
+          python setup.py sdist
+          twine upload dist/*

.github/workflows/ruff-format.yml

@@ -0,0 +1,17 @@
+name: Ruff Formatting
+
+on: push
+
+jobs:
+  formatting-ruff:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+
+      - name: Code Formatting (App)
+        run: |
+          pip install -r requirements-dev.txt
+          ruff format --check .

.github/workflows/ruff.yml

@@ -0,0 +1,17 @@
+name: Ruff Check
+
+on: push
+
+jobs:
+  ruff-check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+
+      - name: Code Formatting (App)
+        run: |
+          pip install -r requirements-dev.txt
+          ruff check .

.github/workflows/tests.yml

@@ -0,0 +1,18 @@
+name: Code Tests
+
+on: push
+
+jobs:
+  testing:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+
+      - name: Run tests
+        run: |
+          pip install .
+          pip install -r requirements-dev.txt
+          python3 -m pytest .

.github/workflows/windows-tests.yml

@@ -0,0 +1,18 @@
+name: Code Tests (On Windows)
+
+on: push
+
+jobs:
+  testing-windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.11.1
+
+      - name: Run tests
+        run: |
+          pip install .
+          pip install -r requirements-dev.txt
+          python3 -m pytest . -s

.gitignore

@@ -0,0 +1,12 @@
+# Mac stuff
+.DS_Store
+
+# Python
+__pycache__/
+dist/
+build/
+.ruff_cache/
+albero.egg-info/
+
+# Pycharm
+.idea

.readthedocs.yaml

@@ -0,0 +1,26 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+  commands:
+    - python3 ./docs/source/create_example_docs.py
+    - python3 -m pip install -r docs/requirements.txt
+    - python3 -m sphinx --builder dirhtml docs/source $READTHEDOCS_OUTPUT/html
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+
+# We recommend specifying your dependencies to enable reproducible builds:
+# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+  - requirements: docs/requirements.txt

README.md

@@ -1,2 +1,27 @@
-# albero
-An tool that makes it easy to use Tree Sitter
+<h1 align="center">Albero v0.0.1</h1>
+
+A tool that makes it much easier to use Tree Sitter for your code editors
+
+# Installation
+
+In the Command Line, paste the following: `pip install albero` (not yet uploaded to pypi)
+
+## Description
+
+Albero is a library that can be used by code editors to easily get syntax highlighting from tree-sitter. It simplifies the process of getting readily usable tokens and only requires a `Language` or `.so` language file along with the languages mapping to get started (some are pre-included with the module).
+
+> **Notes:**
+>  - This project is super early on and will probably be moved to `salve-org` shortly. This originally was created for the main tool `Salve` but that may or may not be split into multiple tools dor simplicity's sake.
+
+## Contributing
+
+To contribute, fork the repository, make your changes, and then make a pull request. If you want to add a feature, please open an issue first so it can be discussed. Note that whenever and wherever possible you should try to use stdlib modules rather than external ones.
+
+## Required Python Version: 3.11+
+
+Albero will use the three most recent versions (full releases) going forward and will drop any older versions as new ones come out. This is because I hope to keep this package up to date with modern python versions as they come out instead of being forced to maintain decade old python versions.
+Currently 3.11 is the minimum (instead of 3.10) as this was developed under 3.12 and there are many features that it relies on from this version I want. However, after 3.14 is released, the minimum version will be 3.12 (as would be expected from the plan) and will change accordingly in the future as is described in the plan above.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](./LICENSE).

albero/__init__.py

@@ -0,0 +1,8 @@
+from beartype.claw import beartype_this_package
+
+beartype_this_package()
+
+from .languages import get_lang, get_mapping  # noqa: F401, E402
+from .misc import AlberoException, lang_from_so  # noqa: F401, E402
+from .tokens import Token, generic_tokens  # noqa: F401, E402
+from .tree_sitter import TreeSitterHighlighter  # noqa: F401, E402

albero/languages/__init__.py

@@ -0,0 +1 @@
+from .compile_languages import get_lang, get_mapping  # noqa: F401

albero/languages/arduino.py

@@ -0,0 +1 @@
+# from tree_sitter_arduino import language as arduino_language (issue opened https://github.com/tree-sitter-grammars/tree-sitter-arduino/issues/30)

albero/languages/bash.py

@@ -0,0 +1 @@
+from tree_sitter_bash import language as bash_language  # noqa: F401

albero/languages/c.py

@@ -0,0 +1 @@
+from tree_sitter_c import language as c_language  # noqa: F401

albero/languages/c_sharp.py

@@ -0,0 +1 @@
+from tree_sitter_c_sharp import language as c_sharp_language  # noqa: F401

albero/languages/commonlisp.py

@@ -0,0 +1,3 @@
+from tree_sitter_commonlisp import (  # noqa: F401
+    language as commonlisp_language,
+)

albero/languages/compile_languages.py

@@ -0,0 +1,85 @@
+from beartype.typing import Callable
+from tree_sitter import Language
+
+from .bash import bash_language
+from .c import c_language
+from .c_sharp import c_sharp_language
+from .commonlisp import commonlisp_language
+from .cpp import cpp_language
+from .css import css_language
+from .cuda import cuda_language
+from .embedded_template import embedded_template_language
+from .glsl import glsl_language
+from .go import go_language
+from .gstlaunch import gstlaunch_language
+from .html import html_language
+from .java import java_language, java_mapping
+from .javascript import javascript_language
+from .jsdoc import jsdoc_language
+from .json import json_language
+from .odin import odin_language
+from .php import php_language
+from .pymanifest import pymanifest_language
+from .python import python_language, python_mapping
+from .ruby import ruby_language
+from .rust import rust_language
+from .scss import scss_language
+from .slang import slang_language
+from .starlark import starlark_language
+from .toml import toml_language
+from .typescript import typescript_language
+from .wgsl_bevy import wgsl_bevy_language
+from .yaml import yaml_language
+
+# from .arduino import arduino_language
+
+language_functions: dict[str, Callable] = {
+    "starlark": starlark_language,
+    "commonlisp": commonlisp_language,
+    "odin": odin_language,
+    "glsl": glsl_language,
+    "javascript": javascript_language,
+    "python": python_language,
+    "toml": toml_language,
+    "bash": bash_language,
+    "c_sharp": c_sharp_language,
+    "c": c_language,
+    "php": php_language,
+    "cuda": cuda_language,
+    "pymanifest": pymanifest_language,
+    # "arduino": arduino_language,
+    "css": css_language,
+    "embedded_template": embedded_template_language,
+    "jsdoc": jsdoc_language,
+    "wgsl_bevy": wgsl_bevy_language,
+    "html": html_language,
+    "yaml": yaml_language,
+    "cpp": cpp_language,
+    "slang": slang_language,
+    "ruby": ruby_language,
+    "java": java_language,
+    "scss": scss_language,
+    "go": go_language,
+    "json": json_language,
+    "rust": rust_language,
+    "typescript": typescript_language,
+    "gstlaunch": gstlaunch_language,
+}
+language_mappings: dict[str, dict[str, str]] = {
+    "python": python_mapping,
+    "java": java_mapping,
+}
+
+
+def get_lang(language_name: str) -> Language:
+    if language_name not in language_functions:
+        raise Exception("Language not in pre-compiled languages")
+
+    return Language(language_functions[language_name]())
+
+
+def get_mapping(language_name: str) -> dict[str, str]:
+    if language_name not in language_mappings:
+        raise Exception("Language not in pre-compiled mappings")
+
+    return language_mappings[language_name]

albero/languages/cpp.py

@@ -0,0 +1 @@
+from tree_sitter_cpp import language as cpp_language  # noqa: F401

albero/languages/css.py

@@ -0,0 +1 @@
+from tree_sitter_css import language as css_language  # noqa: F401

albero/languages/cuda.py

@@ -0,0 +1 @@
+from tree_sitter_cuda import language as cuda_language  # noqa: F401

albero/languages/embedded_template.py

@@ -0,0 +1,3 @@
+from tree_sitter_embedded_template import (  # noqa: F401
+    language as embedded_template_language,
+)

albero/languages/glsl.py

@@ -0,0 +1 @@
+from tree_sitter_glsl import language as glsl_language  # noqa: F401

albero/languages/go.py

@@ -0,0 +1 @@
+from tree_sitter_go import language as go_language  # noqa: F401

albero/languages/gstlaunch.py

@@ -0,0 +1 @@
+from tree_sitter_gstlaunch import language as gstlaunch_language  # noqa: F401

albero/languages/html.py

@@ -0,0 +1 @@
+from tree_sitter_html import language as html_language  # noqa: F401