commit 379751ea9afb8e192fb2dc76edb8b6c9146d4aa0
parent b161c6a9e8d0fabc3ce8baeda571ee1ead6c861e
Author: Andrew D. Laack <andrew@laack.co>
Date: Thu, 19 Jun 2025 20:27:46 -0500
Merge pull request #2 from andrewlaack/toml-naming-change (all of this code was written after my trial project ended)
It should be noted that all of this code in this PR was written outside of my trial project. I'm motivated to finish this because I think it could be useful.
Diffstat:
16 files changed, 588 insertions(+), 293 deletions(-)
diff --git a/README.md b/README.md
@@ -1,26 +1,37 @@
# Ratchets
-Tests that lazily enforce a requirement across the entire repo.
+Tests that lazily enforce a requirement across the entire repository.
# What is it?
-Ratchets is a lazy way to enforce code compliance on an ongoing basis. This is done by defining regular expressions or commands to run against all non-excluded python files in a given repository. Tests pass when the number of non-compliant instances of code decreases and fail when they increase. This ensures that subsequent code does not have bad patterns, while still allowing old code to coexist until it is phased out.
+Ratchets is a lazy way to enforce code compliance on an ongoing basis. This is done by defining regular expressions and shell commands to run against all non-excluded python files in a given repository. Tests pass when the number of non-compliant instances of code decreases and fail when they increase. This ensures future code does not have bad patterns, while still allowing old code to coexist until it is phased out.
# Installation
+## Required
+
```bash
pip install ratchets
```
+## Optional
+
+This is only required if you plan to use Ratchets with PyTest.
+
+```bash
+pip install pytest
+```
+
# Usage
-You first need to create a tests.toml file at the root of your repository. See [tests.toml](tests.toml) for an example of how this should look. There are two primary rule types that can be defined in the tests.toml file.
+First, create a tests.toml file at the root of your repository. See [tests.toml](https://github.com/andrewlaack/ratchets/blob/toml-naming-change/tests.toml) for an example of how this should look. There are two primary rule types that can be defined in the tests.toml file.
## ratchet.regex
-These are tests that check regular expressions on the basis of each line of each file being examined.
+These are tests that check regular expressions for each line of code in each file being examined.
**Example:**
+
```toml
[ratchet.regex.exceptions]
@@ -49,29 +60,29 @@ except:
```
-The valid and invalid entries are not necessary, but we provide a CLI utility, ran with ```python3 -m ratchets.validate```, to verify the regular expressions don't exist in the valid string and do exist in the invalid string. If you are testing the tests.toml file in the current git repository or ```python3 -m ratchets.validate -f FILENAME``` if you need to test a specific toml file.
-
+The valid and invalid entries are not necessary, but we provide a CLI utility, executable with ```python3 -m ratchets.validate```, to verify the regular expressions don't exist in the valid string and do exist in the invalid string. If you are testing a .toml file that is not the repository default, specify it with ```python3 -m ratchets.validate -f FILENAME```.
## ratchet.shell
These are tests that run against each file where each evaluation is of the form:
```bash
-FILEPATH | COMMAND
+FILEPATH | SHELL_COMMAND
```
-It is assumed the standard output of the command describes each of the issues where each line is counted as an infraction.
+
+The standard output of the command is assumed to describe infractions, and the number of lines dictates the total number of infractions. It should also be noted that internally we perform a lookup for the line number based on the standard output. As such, ensure the standard output is the **exact** same text from the line that contains infractions.
**Example:**
```toml
[ratchet.shell.line_too_long]
-command = "xargs -n1 awk 'length($0) > 80'"
+command = "xargs -n1 awk 'length($0) > 88'"
```
-This is an example of an `awk` command being used to print each line that has more than 80 characters. As these are printed, they are counted as infractions.
+This is an example of an `awk` command being used to print each line that has more than 88 characters (the default for [black](https://github.com/psf/black). As these are printed, they are counted as infractions.
## Updating Ratchets
@@ -81,61 +92,62 @@ Once your rules are defined, you need to count the infractions. This is done by
python3 -m ratchets -u
```
-This creates a ratchet_values.json file in the root of your project. This will be checked into git and is how the previous number of infractions is tracked to ensure infraction counts never increase.
+This creates a ratchet_values.json file in the root of your project. This should be checked into git to manage state.
## Excluding Files
-Once you run the update command, you should see a file in the root of your repository titled `ratchet_excluded.txt`. By default, this file is empty, but you can use standard .gitignore syntax to specify files that shouldn't be included in your tests. Additionally, all files specified by the gitignore of your project or that don't have the .py extension will not be included in the evaluation.
+Once the update command has been executed, the `ratchet_excluded.txt` file is created at the root of the repository. By default, this file is empty, but standard .gitignore syntax can be used to specify files that shouldn't be included in tests. Additional files that won't be tested are files specified in your gitignore and files that don't have the extension .py.
## Running as part of PyTest
-To set up tests, we provide an example file at [examples/example_test_ratchet.py](examples/example_test_ratchet.py), which defines tests to be ran with PyTest. In this file there are two uncommented methods that runs one test per rule in both sections (Python and command).
+To set up tests, we provide an example file at [examples/example_test_ratchet.py](https://github.com/andrewlaack/ratchets/blob/toml-naming-change/examples/example_test_ratchet.py), which defines tests to be ran with PyTest. In this file there are two uncommented methods that runs one test per rule in both sections (regex and shell).
-The commented methods aggregate these tests together into two total tests (Python and command).
+The commented methods aggregate these tests together into two total tests (regex and shell).
-When creating your testing file, ensure it is being indexed by pytest. If you are unsure what this means, create a file named `test_ratchet.py` in the root of your project.
+When creating your PyTest file, ensure it is being indexed by PyTest. If you are unsure what this means, create a file named `test_ratchet.py` in the root of your project.
## Running Tests
Running tests is as simple as running ```pytest``` from the root of the repository or specifying the testing file with ```pytest test_ratchet.py```.
-## Finding Issues
+## Additional Functionality
-At this point, your project has been set up, but as these tests are ran, and further infringements are found, there is a need to identify them. This, along with many other pieces of functionality can be viewed by running:
+Beyond a seamless integration with PyTest, Ratchets provides functionality to find the location of infringements. This and other functionality can be found by running:
```
python3 -m ratchets --help
```
-Where you will see the following help message describing CLI usage for ratchets:
+Where you will see the following help message describing CLI usage for Ratchets:
```
-usage: __main__.py [-h] [-f FILE] [-c] [-r] [-v] [-b] [-m MAX_COUNT] [--compare-counts] [-u]
+usage: run_tests.py [-h] [-t TOML_FILE] [-f FILES [FILES ...]] [-s] [-r] [-v] [-b] [-m MAX_COUNT] [-c] [-u]
Python ratchet testing
options:
-h, --help show this help message and exit
- -f FILE, --file FILE specify .toml file with tests
- -c, --command-only run only custom command-based tests
+ -t TOML_FILE, --toml-file TOML_FILE
+ specify a .toml file with tests
+ -f FILES [FILES ...], --files FILES [FILES ...]
+ specify file(s) to evaluate
+ -s, --shell-only run only shell-based tests
-r, --regex-only run only regex-based tests
-v, --verbose run verbose tests, printing each infringing line
-b, --blame run an additional git-blame for each infraction, ordering results by timestamp
-m MAX_COUNT, --max-count MAX_COUNT
maximum infractions to display per test (only applies with --blame; default is 10)
- --compare-counts show only the differences in infraction counts between the current and last saved tests
+ -c, --compare-counts show only the differences in infraction counts between the current and last saved tests
-u, --update-ratchets
update ratchets_values.json
```
-Of these, the -b option is particularly useful. When PyTests fail due to infringement counts increasing, it is necessary to identify where the new infringement occurred. By using the -b option you will, by default, see the 10 most recent changes that caused infringements for each rule.
-
# Testing Ratchets Locally
-To run the tests for the Ratchets source code locally you can clone this repository with:
+To run the tests for the source code of Ratchets, you can clone this repository with:
```bash
git clone https://github.com/andrewlaack/ratchets/
```
-Then `cd` into ratchets and run `PyTest`. The tests use the installed version of Ratchets in your (virtual) environment so you must ensure changes to source files are applied to Ratchets there.
+Then `cd` into `ratchets` and run `PyTest`. The tests use the installed version of Ratchets from your virtual environment. This means you must ensure changes to source files are applied to your installed `ratchets` package prior to running the tests.
diff --git a/examples/example_test_ratchet.py b/examples/example_test_ratchet.py
@@ -1,35 +1,46 @@
import pytest
-from ratchets.abstracted_tests import get_python_tests, get_command_tests, \
- check_python_rule, check_command_rule
+from ratchets.abstracted_tests import (
+ get_regex_tests,
+ get_shell_tests,
+ check_regex_rule,
+ check_shell_rule,
+)
-@pytest.mark.parametrize("test_name,rule", get_python_tests().items())
-def test_python_regex_rule(test_name: str, rule: dict) -> None:
- check_python_rule(test_name, rule)
-@pytest.mark.parametrize("test_name,test_dict", get_command_tests().items())
-def test_custom_command_rule(test_name: str, test_dict: dict) -> None:
- check_command_rule(test_name, test_dict)
+@pytest.mark.parametrize("test_name,rule", get_regex_tests().items())
+def test_regex_rule(test_name: str, rule: dict) -> None:
+ """Runs a test for a single regex rule."""
+ check_regex_rule(test_name, rule)
-# def test_all_python_regex_rules():
+
+@pytest.mark.parametrize("test_name,test_dict", get_shell_tests().items())
+def test_shell_rule(test_name: str, test_dict: dict) -> None:
+ """Runs a test for a single shell rule."""
+ check_shell_rule(test_name, test_dict)
+
+
+# def test_all_regex_rules():
+# """Runs a test for all regex rules."""
# errors = []
-# for test_name, rule in get_python_tests().items():
+# for test_name, rule in get_regex_tests().items():
# try:
-# check_python_rule(test_name, rule)
-# except AssertionError as e:
+# check_regex_rule(test_name, rule)
+# except Exception as e:
# errors.append(f"{test_name}: {e}")
# except Exception as e:
# errors.append(f"{test_name}: unexpected error: {e!r}")
# if errors:
-# pytest.fail("Some python regex rules failed:\n" + "\n".join(errors))
-#
-# def test_all_command_rules():
+# pytest.fail("Some regex rules failed:\n" + "\n".join(errors))
+#
+# def test_all_shell_rules():
+# """Runs a test for all shell rules."""
# errors = []
-# for test_name, test_dict in get_command_tests().items():
+# for test_name, test_dict in get_shell_tests().items():
# try:
-# check_command_rule(test_name, test_dict)
-# except AssertionError as e:
+# check_shell_rule(test_name, test_dict)
+# except Exception as e:
# errors.append(f"{test_name}: {e}")
# except Exception as e:
# errors.append(f"{test_name}: unexpected error: {e!r}")
# if errors:
-# pytest.fail("Some command rules failed:\n" + "\n".join(errors))
+# pytest.fail("Some shell rules failed:\n" + "\n".join(errors))
diff --git a/pyproject.toml b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ratchets"
-version = "0.1.2"
+version = "0.1.3"
description = "Ratcheted testing in Python."
authors = [
{ name = "Andrew Laack", email = "andrew@laack.co" }
diff --git a/src/ratchets/__main__.py b/src/ratchets/__main__.py
@@ -1,3 +1,4 @@
from .run_tests import cli
+
if __name__ == "__main__":
cli()
diff --git a/src/ratchets/abstracted_tests.py b/src/ratchets/abstracted_tests.py
@@ -5,8 +5,8 @@ from pathlib import Path
from typing import Dict, Any, List
from .run_tests import (
- evaluate_python_tests,
- evaluate_command_tests,
+ evaluate_regex_tests,
+ evaluate_shell_tests,
filter_excluded_files,
find_project_root,
get_python_files,
@@ -20,7 +20,7 @@ def get_root() -> str:
def get_config() -> Dict[str, Any]:
- """Load and return the tests.toml configuration as a dict."""
+ """Load and return the tests.toml configuration."""
root = get_root()
toml_path = Path(root) / "tests.toml"
try:
@@ -29,27 +29,26 @@ def get_config() -> Dict[str, Any]:
return {}
-def get_python_tests() -> Dict[str, Any]:
- """Extract and return the 'python-tests' section from config."""
+def get_regex_tests() -> Dict[str, Any]:
+ """Extract and return the 'ratchet.regex' section from config."""
config = get_config()
python_tests = config.get("ratchet", {}).get("regex")
return python_tests or {}
-def get_command_tests() -> Dict[str, Any]:
+def get_shell_tests() -> Dict[str, Any]:
"""Extract and return the 'ratchet.shell' section from config."""
config = get_config()
shell_tests = config.get("ratchet", {}).get("shell")
return shell_tests or {}
-
def load_baseline_counts() -> Dict[str, int]:
- """Load baseline counts from ratchet path, returning a dict of test_name to count."""
+ """Load baseline counts from ratchet path, returning a dict of test names and counts."""
try:
ratchet_path: str = get_ratchet_path()
if os.path.isfile(ratchet_path):
- with open(ratchet_path, 'r', encoding='utf-8') as f:
+ with open(ratchet_path, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, dict):
return {k: int(v) for k, v in data.items()}
@@ -59,40 +58,49 @@ def load_baseline_counts() -> Dict[str, int]:
def get_baseline_counts() -> Dict[str, int]:
- """Return baseline counts, caching on first call."""
+ """Return baseline counts"""
return load_baseline_counts()
def get_filtered_files() -> List[Path]:
"""Retrieve all Python files under the project, filtering excluded paths."""
root = get_root()
- files: List[Path] = get_python_files(root)
+ files: List[Path] = get_python_files(root, None)
excluded_path: str = os.path.join(root, "ratchet_excluded.txt")
ignore_path: str = os.path.join(root, ".gitignore")
+
try:
return filter_excluded_files(files, excluded_path, ignore_path)
except Exception:
return files
-def get_python_test_matches(test_name: str, rule: Dict[str, Any]) -> List[Dict[str, Any]]:
- """Run the Python regex test for a single rule and return matches."""
+def get_python_test_matches(
+ test_name: str, rule: Dict[str, Any]
+) -> List[Dict[str, Any]]:
+ """Run a regex test for a single rule and return matches."""
files = get_filtered_files()
- results: Dict[str, List[Dict[str, Any]]] = evaluate_python_tests(files, {test_name: rule})
+ results: Dict[str, List[Dict[str, Any]]] = evaluate_regex_tests(
+ files, {test_name: rule}
+ )
return results.get(test_name, [])
-def get_command_test_matches(test_name: str, test_dict: Dict[str, Any]) -> List[Dict[str, Any]]:
- """Run the custom command test for a single rule and return matches."""
+def get_shell_test_matches(
+ test_name: str, test_dict: Dict[str, Any]
+) -> List[Dict[str, Any]]:
+ """Run a shell test for a single rule and return matches."""
files = get_filtered_files()
- results: Dict[str, List[Dict[str, Any]]] = evaluate_command_tests(files, {test_name: test_dict})
+ results: Dict[str, List[Dict[str, Any]]] = evaluate_shell_tests(
+ files, {test_name: test_dict}
+ )
return results.get(test_name, [])
-def check_python_rule(test_name: str, rule: Dict[str, Any]) -> None:
-
- assert (test_name is not None)
- assert (rule is not None)
+def check_regex_rule(test_name: str, rule: Dict[str, Any]) -> None:
+ """Check if a single regex rule has been violated by increasing infraction count."""
+ assert test_name is not None
+ assert rule is not None
matches = get_python_test_matches(test_name, rule)
current_count = len(matches)
@@ -102,24 +110,24 @@ def check_python_rule(test_name: str, rule: Dict[str, Any]) -> None:
details = "\n".join(
f"{r.get('file')}:{r.get('line')} — {r.get('content')}" for r in matches
)
- raise AssertionError(
- f"Regex violations for '{test_name}' increased: baseline={baseline_count}, current={current_count}\n" + details
+ raise Exception(
+ f"Regex violations for '{test_name}' increased: baseline={baseline_count}, current={current_count}\n"
+ + details
)
-def check_command_rule(test_name: str, test_dict: Dict[str, Any]) -> None:
-
- assert (test_name is not None)
- assert (test_dict is not None)
+def check_shell_rule(test_name: str, test_dict: Dict[str, Any]) -> None:
+ """Check if a single shell rule has been violated by increasing infraction count."""
+ assert test_name is not None
+ assert test_dict is not None
- matches = get_command_test_matches(test_name, test_dict)
+ matches = get_shell_test_matches(test_name, test_dict)
current_count = len(matches)
baseline_counts = get_baseline_counts()
baseline_count = baseline_counts.get(test_name, 0)
if current_count > baseline_count:
- details = "\n".join(
- f"{r.get('file')} — {r.get('content')}" for r in matches
- )
- raise AssertionError(
- f"Command violations for '{test_name}' increased: baseline={baseline_count}, current={current_count}\n" + details
+ details = "\n".join(f"{r.get('file')} — {r.get('content')}" for r in matches)
+ raise Exception(
+ f"shell violations for '{test_name}' increased: baseline={baseline_count}, current={current_count}\n"
+ + details
)
diff --git a/src/ratchets/bad.py b/src/ratchets/bad.py
@@ -1,5 +0,0 @@
-except:
-except:
-except:
-except:
-except:
diff --git a/src/ratchets/run_tests.py b/src/ratchets/run_tests.py
@@ -10,10 +10,17 @@ import re
import subprocess
from typing import Optional, List, Dict, Tuple, Union, Any
+EXCLUDED_FILENAME = "ratchet_excluded.txt"
+IGNORE_FILENAME = ".gitignore"
+RATCHET_FILENAME = "ratchet_values.json"
+TEST_FILENAME = "tests.toml"
+
def print_diff(current_json: Dict[str, int], previous_json: Dict[str, int]) -> None:
+ """Print formatted json and differences."""
all_keys = set(current_json.keys()) | set(previous_json.keys())
diff_count = 0
+
for key in sorted(all_keys):
current_value = current_json.get(key, 0)
previous_value = previous_json.get(key, 0)
@@ -21,17 +28,24 @@ def print_diff(current_json: Dict[str, int], previous_json: Dict[str, int]) -> N
diff_count += 1
diff = current_value - previous_value
sign = "+" if diff > 0 else "-"
- print(f" {key}: {previous_value} → {current_value} ({sign}{abs(diff)})")
+ print(f" {key}: {previous_value} -> {current_value} ({sign}{abs(diff)})")
+
if diff_count == 0:
print("There are no differences.")
-def find_project_root(start_path: Optional[str] = None, markers: Optional[List[str]] = None) -> str:
+def find_project_root(
+ start_path: Optional[str] = None, markers: Optional[List[str]] = None
+) -> str:
+ """Return the root of the current project starting from start_path or cwd."""
if start_path is None:
start_path = os.getcwd()
+
if markers is None:
- markers = ['.git', 'pyproject.toml', 'setup.py', 'tests.toml']
+ markers = [".git", "pyproject.toml", "setup.py", "tests.toml"]
+
current = os.path.abspath(start_path)
+
while True:
for marker in markers:
if os.path.exists(os.path.join(current, marker)):
@@ -43,94 +57,127 @@ def find_project_root(start_path: Optional[str] = None, markers: Optional[List[s
def get_excludes_path() -> str:
- DEFAULT_FILENAME = "ratchet_excluded.txt"
+ """Get the path for the 'ratchet_excluded.txt' file."""
root = find_project_root(None)
- return os.path.join(root, DEFAULT_FILENAME)
+ return os.path.join(root, EXCLUDED_FILENAME)
def get_file_path(file: Optional[str]) -> str:
- DEFAULT_FILENAME = "tests.toml"
- if not file:
- file = DEFAULT_FILENAME
- if "/" in file:
- return file
- else:
+ """Get the path, as a string, for the 'tests.toml' file or return a file path with a matching name to 'file'."""
+ if file is None or len(file) == 0:
+ file = TEST_FILENAME
root = find_project_root(file)
- return os.path.join(root, file)
+ return str(os.path.join(root, file))
+ return file
+
+def get_python_files(
+ directory: Union[str, Path], paths: Optional[List[str]]
+) -> List[Path]:
+ """Return a list of paths for python files in the specified directory."""
+
+ if paths:
+ path_paths = [Path(x) for x in paths]
+ return path_paths
-def get_python_files(directory: Union[str, Path]) -> List[Path]:
directory = Path(directory)
- python_files = set([path.absolute() for path in directory.rglob("*.py") if not path.is_symlink()])
+ python_files = set(
+ [path.absolute() for path in directory.rglob("*.py") if not path.is_symlink()]
+ )
return list(python_files)
-def filter_excluded_files(files: List[Path], excluded_path: str, ignore_path: str) -> List[Path]:
- with open(excluded_path, 'r') as f:
- patterns = f.read().splitlines()
+def filter_excluded_files(
+ files: List[Path], excluded_path: str, ignore_path: str
+) -> List[Path]:
+ """Returns a new list of file paths that consists of all 'files' paths not excluded in the 'excluded_path' or 'ignore_path'."""
+ patterns = []
+ if os.path.isfile(excluded_path):
+ with open(excluded_path, "r") as f:
+ patterns += f.read().splitlines()
+
if os.path.isfile(ignore_path):
- with open(ignore_path, 'r') as f:
+ with open(ignore_path, "r") as f:
patterns += f.read().splitlines()
- spec = pathspec.PathSpec.from_lines('gitwildmatch', patterns)
+
+ spec = pathspec.PathSpec.from_lines("gitwildmatch", patterns)
files = [f for f in files if not spec.match_file(f)]
return files
-def evaluate_tests(path: str, cmd_only: bool, regex_only: bool) -> Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]:
-
+def evaluate_tests(
+ path: str,
+ cmd_only: bool,
+ regex_only: bool,
+ paths: Optional[List[str]],
+ override_filter: bool = False,
+) -> Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]:
+ """Runs all requested tests based on the 'path' .toml file."""
assert os.path.isfile(path)
config = toml.load(path)
- python_tests = config.get("ratchet", {}).get("regex")
- custom_tests = config.get("ratchet", {}).get("shell")
+ regex_tests = config.get("ratchet", {}).get("regex")
+ shell_tests = config.get("ratchet", {}).get("shell")
+
root = find_project_root()
- files = get_python_files(root)
- EXCLUDED_PATH = "ratchet_excluded.txt"
- excluded_path = os.path.join(root, EXCLUDED_PATH)
- ignore_path = os.path.join(root, ".gitignore")
- files = filter_excluded_files(files, excluded_path, ignore_path)
- test_issues: Dict[str, List[Dict[str, Any]]] = {}
- custom_issues: Dict[str, List[Dict[str, Any]]] = {}
- if python_tests and not cmd_only:
- test_issues = evaluate_python_tests(files, python_tests)
- if custom_tests and not regex_only:
- custom_issues = evaluate_command_tests(files, custom_tests)
- return test_issues, custom_issues
+ files = get_python_files(root, paths)
+
+ excluded_path = os.path.join(root, EXCLUDED_FILENAME)
+ ignore_path = os.path.join(root, IGNORE_FILENAME)
+
+ if not override_filter:
+ files = filter_excluded_files(files, excluded_path, ignore_path)
+
+ regex_issues: Dict[str, List[Dict[str, Any]]] = {}
+ shell_issues: Dict[str, List[Dict[str, Any]]] = {}
+
+ if regex_tests and not cmd_only:
+ regex_issues = evaluate_regex_tests(files, regex_tests)
+ if shell_tests and not regex_only:
+ shell_issues = evaluate_shell_tests(files, shell_tests)
+ return regex_issues, shell_issues
def print_issues(issues: Dict[str, List[Dict[str, Any]]]) -> None:
+ """Print the 'issues' dict in a human readable way."""
for test_name, matches in issues.items():
if matches:
print(f"\n{test_name} — matched {len(matches)} issue(s):")
for match in matches:
- file_path = match['file']
- line = match.get('line')
- content = match['content']
- truncated = content if len(content) <= 80 else content[:80] + "..."
+ file_path = match["file"]
+ line = match.get("line")
+ content = match["content"]
+ truncated = content if len(content) <= 50 else content[:50] + "..."
if line is not None:
- print(f" → {file_path}:{line}: {truncated}")
+ print(f" -> {file_path}:{line}: {truncated}")
else:
- print(f" → {file_path}: {truncated}")
+ print(f" -> {file_path}: {truncated}")
else:
print(f"\n{test_name} — no issues found.")
def load_ratchet_results() -> Dict[str, Any]:
-
+ """Load and return current ratchet values.."""
path = get_ratchet_path()
if not os.path.isfile(path):
return {}
- with open(path, 'r') as file:
+ with open(path, "r") as file:
data = json.load(file)
- return data
+
+ return dict(data)
-def evaluate_python_tests(files: List[Path], test_str: Dict[str, Dict[str, Any]]) -> Dict[str, List[Dict[str, Any]]]:
- assert len(files) != 0
- assert len(test_str) != 0
+def evaluate_regex_tests(
+ files: List[Path], test_str: Dict[str, Dict[str, Any]]
+) -> Dict[str, List[Dict[str, Any]]]:
+ """Evaluate a list of regex tests in parallel with one thread per test."""
+ if len(files) == 0:
+ raise Exception("No files were passed in to be evaluated.")
+ if len(test_str) == 0:
+ raise Exception("No regex tests were passed in to be evaluated.")
results: Dict[str, List[Dict[str, Any]]] = {}
threads = []
@@ -141,14 +188,16 @@ def evaluate_python_tests(files: List[Path], test_str: Dict[str, Dict[str, Any]]
matches = []
for file_path in files:
- with open(file_path, 'r', encoding='utf-8') as f:
+ with open(file_path, "r", encoding="utf-8") as f:
for lineno, line in enumerate(f, 1):
if pattern.search(line):
- matches.append({
- "file": str(file_path),
- "line": lineno,
- "content": line.strip()
- })
+ matches.append(
+ {
+ "file": str(file_path),
+ "line": lineno,
+ "content": line.strip(),
+ }
+ )
with results_lock:
results[test_name] = matches
@@ -164,48 +213,94 @@ def evaluate_python_tests(files: List[Path], test_str: Dict[str, Dict[str, Any]]
def get_ratchet_path() -> str:
+ """Get the path for the ratchet values file on disk."""
root = find_project_root()
- RATCHET_NAME = "ratchet_values.json"
- ratchet_file_path = os.path.join(root, RATCHET_NAME)
+ ratchet_file_path = os.path.join(root, RATCHET_FILENAME)
return ratchet_file_path
-def evaluate_command_tests(files: List[Path], test_str: Dict[str, Dict[str, Any]]) -> Dict[str, List[Dict[str, Any]]]:
- assert len(test_str) != 0
- assert len(files) != 0
+def evaluate_shell_tests(
+ files: List[Path], test_str: Dict[str, Dict[str, Any]]
+) -> Dict[str, List[Dict[str, Any]]]:
+ """Evaluate all shell tests in parallel across each file."""
+ if len(test_str) == 0:
+ raise Exception("No shell tests passed to evaluation method.")
+ if len(files) == 0:
+ raise Exception("No files passed to evaluation method.")
results: Dict[str, List[Dict[str, Any]]] = {test_name: [] for test_name in test_str}
lock = threading.Lock()
- def worker(test_name: str, command_template: str, file_path: str):
- cmd_str = f"echo {file_path} | {command_template}"
+ # we track each line number
+ # for duplicates, popping them
+ # as they are used, if they are
+ # used.
+
+ file_lines_map: Dict[str, Dict[str, List[int]]] = {}
+
+ for file_path in files:
+ try:
+ with open(file_path, "r", encoding="utf-8") as f:
+ lines = f.readlines()
+ file_map: Dict[str, List[int]] = {}
+ for idx, line in enumerate(lines):
+ normalized = line.rstrip("\n")
+ file_map.setdefault(normalized, []).append(idx + 1)
+ file_lines_map[str(file_path)] = file_map
+ except Exception as e:
+ raise Exception(f"Error reading {file_path}: {e}")
+
+ def worker(test_name: str, shell_template: str, file_path: Path):
+ """Evaluate an individual shell test for a given file."""
+ file_str = str(file_path)
+ cmd_str = f"echo {file_str} | {shell_template}"
+
try:
result = subprocess.run(
- cmd_str,
- shell=True,
- text=True,
- capture_output=True,
- timeout=5
+ cmd_str, shell=True, text=True, capture_output=True, timeout=5
)
+
output = result.stdout.strip()
+
if output:
lines = output.splitlines()
+
with lock:
for line in lines:
- results[test_name].append({
- "file": file_path,
- "line": None,
- "content": line.strip()
- })
+
+ content = line.rstrip("\n")
+ line_numbers = file_lines_map[file_str].get(content, [])
+
+ # assume we found the last line this happened,
+ # remove it, and repeat for each infraction of this line.
+ # this can be wrong, but it is impossible to
+ # solve the ambiguity of multiple lines matching,
+ # but not causing infractions, which can happen
+ # when shell commands are defined to consider multiple lines,
+ # as can be the case with ast and such.
+
+ if line_numbers:
+ ln = line_numbers[0]
+ line_numbers.pop()
+ results[test_name].append(
+ {
+ "file": file_str,
+ "line": ln,
+ "content": content,
+ }
+ )
+
except subprocess.TimeoutExpired:
- print(f"Timeout while running test '{test_name}' on {file_path}")
+ raise Exception(f"Timeout while running test '{test_name}' on {file_path}")
threads = []
for test_name, test_dict in test_str.items():
- command_template = test_dict["command"]
+ shell_template = test_dict["command"]
for file_path in files:
- t = threading.Thread(target=worker, args=(test_name, command_template, file_path))
+ t = threading.Thread(
+ target=worker, args=(test_name, shell_template, file_path)
+ )
t.start()
threads.append(t)
@@ -215,28 +310,42 @@ def evaluate_command_tests(files: List[Path], test_str: Dict[str, Dict[str, Any]
return results
-def results_to_json(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]) -> str:
- test_issues, custom_issues = results
+def results_to_json(
+ results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]],
+) -> str:
+ """Convert test results to a standard JSON formatted string."""
+ test_issues, shell_issues = results
counts: Dict[str, int] = {}
+
for name, matches in test_issues.items():
counts[name] = len(matches)
- for name, matches in custom_issues.items():
+
+ for name, matches in shell_issues.items():
counts[name] = counts.get(name, 0) + len(matches)
+
return json.dumps(counts, indent=2, sort_keys=True)
-def update_ratchets(test_path: str, cmd_mode: bool, regex_mode: bool) -> None:
- results = evaluate_tests(test_path, cmd_mode, regex_mode)
+def update_ratchets(
+ test_path: str, cmd_mode: bool, regex_mode: bool, paths: Optional[List[str]]
+) -> None:
+ """Update the current ratchets based on the outcome of the tests defined in 'test_path'."""
+ results = evaluate_tests(test_path, cmd_mode, regex_mode, paths)
results_json = results_to_json(results)
path = get_ratchet_path()
- with open(path, 'w') as file:
+ with open(path, "w") as file:
file.writelines(results_json)
-def print_issues_with_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]], max_count: int) -> None:
- enriched_test_issues, enriched_custom_issues = add_blames(results)
+def print_issues_with_blames(
+ results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]],
+ max_count: int,
+) -> None:
+ """For the results in 'results', get blame results for each result and then print the results in a human readable format."""
+ enriched_test_issues, enriched_shell_issues = add_blames(results)
def _parse_time(ts: Optional[str]) -> datetime:
+ """Internal method used to convert formatted strings to datetimes."""
if not ts:
return datetime.max
try:
@@ -244,60 +353,84 @@ def print_issues_with_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dic
except Exception:
return datetime.max
- def _print_section(section_name: str, issues_dict: Dict[str, List[Dict[str, Any]]]) -> None:
+ def _print_section(
+ section_name: str, issues_dict: Dict[str, List[Dict[str, Any]]]
+ ) -> None:
+ """Internal method used to print the results from an individual test."""
for test_name, matches in issues_dict.items():
if matches:
- sorted_matches = sorted(matches, key=lambda m: _parse_time(m.get("blame_time"))) # type: ignore
- print("\n" + "-" * 40)
- print(f"{section_name} — {test_name} ({len(sorted_matches)} issue{'s' if len(sorted_matches) != 1 else ''}):")
- print("-" * 40)
+ sorted_matches = sorted(
+ matches, key=lambda m: _parse_time(m.get("blame_time"))
+ )
+ print()
+ print(
+ f"{section_name} — {test_name} ({len(sorted_matches)} issue{'s' if len(sorted_matches) != 1 else ''}):"
+ )
+ print()
count = 0
+
for match in sorted_matches:
count += 1
+
if count > max_count:
break
+
file_path = match.get("file", "<unknown>")
line_no = match.get("line")
content = match.get("content", "").strip()
truncated = content if len(content) <= 80 else content[:80] + "..."
author = match.get("blame_author") or "Unknown"
ts = match.get("blame_time") or "Unknown"
+
if line_no is not None:
- print(f" → {file_path}:{line_no} by {author} at {ts}")
+ print(f" -> {file_path}:{line_no} by {author} at {ts}")
print(f" {truncated}")
else:
- print(f" → {file_path} by {author} at {ts}")
+ print(
+ f" -> {file_path} file last updated by {author} at {ts}"
+ )
print(f" {truncated}")
else:
print(f"\n{section_name} — {test_name}: no issues found.")
_print_section("Regex Test", enriched_test_issues)
- _print_section("Command Test", enriched_custom_issues)
+ _print_section("Shell Test", enriched_shell_issues)
+
+def add_blames(
+ results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]],
+) -> Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]:
+ """Add blame information to each result in the 'results' tuple."""
+ test_issues, shell_issues = results
-def add_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]) -> Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Dict[str, Any]]]]:
- test_issues, custom_issues = results
try:
repo_root: Optional[str] = find_project_root()
except Exception:
repo_root = None
- def get_blame_for_line(file_path: str, line_no: Optional[int]) -> Tuple[Optional[str], Optional[str]]:
+ def get_blame_for_line(
+ file_path: str, line_no: Optional[int]
+ ) -> Tuple[Optional[str], Optional[str]]:
+ """Internal method for getting the blame information of a specific LoC."""
if repo_root is None:
return None, None
cmd = ["git", "blame", "-L", f"{line_no},{line_no}", "--porcelain", file_path]
try:
- res = subprocess.run(cmd, capture_output=True, text=True, cwd=repo_root, timeout=5)
+ res = subprocess.run(
+ cmd, capture_output=True, text=True, cwd=repo_root, timeout=5
+ )
if res.returncode != 0:
return None, None
+
author: Optional[str] = None
author_time: Optional[str] = None
+
for l in res.stdout.splitlines():
if l.startswith("author "):
- author = l[len("author "):].strip()
+ author = l[len("author ") :].strip()
elif l.startswith("author-time "):
try:
- ts = int(l[len("author-time "):].strip())
+ ts = int(l[len("author-time ") :].strip())
author_time = datetime.fromtimestamp(ts).isoformat()
except Exception:
author_time = None
@@ -308,11 +441,14 @@ def add_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Di
return None, None
def get_last_commit_for_file(file_path: str) -> Tuple[Optional[str], Optional[str]]:
+ """Internal method to get the most recent commit's information for a file."""
if repo_root is None:
return None, None
cmd = ["git", "log", "-1", "--format=%an;%at", "--", file_path]
try:
- res = subprocess.run(cmd, capture_output=True, text=True, cwd=repo_root, timeout=5)
+ res = subprocess.run(
+ cmd, capture_output=True, text=True, cwd=repo_root, timeout=5
+ )
if res.returncode != 0 or not res.stdout.strip():
return None, None
out = res.stdout.strip()
@@ -329,7 +465,7 @@ def add_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Di
except Exception:
return None, None
- for issues in (test_issues, custom_issues):
+ for issues in (test_issues, shell_issues):
for test_name, matches in issues.items():
for match in matches:
file_path = match.get("file")
@@ -343,102 +479,119 @@ def add_blames(results: Tuple[Dict[str, List[Dict[str, Any]]], Dict[str, List[Di
match["blame_author"] = author if author is not None else None
match["blame_time"] = author_time if author_time is not None else None
- return test_issues, custom_issues
-
-
+ return test_issues, shell_issues
def cli():
+ """Primary entry point for CLI usage, providing parsing and function calls."""
parser = argparse.ArgumentParser(description="Python ratchet testing")
- # Input file
- parser.add_argument("-f", "--file", help="specify .toml file with tests")
+ parser.add_argument("-t", "--toml-file", help="specify a .toml file with tests")
+
+ parser.add_argument("-f", "--files", nargs="+", help="specify file(s) to evaluate")
- # Filtering modes
parser.add_argument(
- "-c", "--command-only",
- action="store_true",
- help="run only custom command-based tests"
+ "-s", "--shell-only", action="store_true", help="run only shell-based tests"
)
+
parser.add_argument(
- "-r", "--regex-only",
- action="store_true",
- help="run only regex-based tests"
+ "-r", "--regex-only", action="store_true", help="run only regex-based tests"
)
- # Output formatting
parser.add_argument(
- "-v", "--verbose",
+ "-v",
+ "--verbose",
action="store_true",
- help="run verbose tests, printing each infringing line"
+ help="run verbose tests, printing each infringing line",
)
- # Blame and related
parser.add_argument(
- "-b", "--blame",
+ "-b",
+ "--blame",
action="store_true",
- help="run an additional git-blame for each infraction, ordering results by timestamp"
+ help="run an additional git-blame for each infraction, ordering results by timestamp",
)
+
parser.add_argument(
- "-m", "--max-count",
+ "-m",
+ "--max-count",
type=int,
- help="maximum infractions to display per test (only applies with --blame; default is 10)"
+ help="maximum infractions to display per test (only applies with --blame; default is 10)",
)
- # Modes of operation
parser.add_argument(
+ "-c",
"--compare-counts",
action="store_true",
- help="show only the differences in infraction counts between the current and last saved tests"
+ help="show only the differences in infraction counts between the current and last saved tests",
)
+
parser.add_argument(
- "-u", "--update-ratchets",
+ "-u",
+ "--update-ratchets",
action="store_true",
- help="update ratchets_values.json"
+ help="update ratchets_values.json",
)
args = parser.parse_args()
- file: Optional[str] = args.file
- cmd_mode: bool = args.command_only
+ file: Optional[str] = args.toml_file
+ cmd_mode: bool = args.shell_only
regex_mode: bool = args.regex_only
update: bool = args.update_ratchets
compare_counts: bool = args.compare_counts
blame: bool = args.blame
verbose: bool = args.verbose
max_count: Optional[int] = args.max_count
+ paths: List[str] = args.files
excludes_path = get_excludes_path()
+ mutex_options = [[cmd_mode, regex_mode], [blame, verbose, update, compare_counts]]
+
+ for ls in mutex_options:
+ if not ls.count(True) <= 1:
+ raise Exception("Mutually exclusive options selected.")
+
if not os.path.isfile(excludes_path):
- with open(excludes_path, 'a'):
+ with open(excludes_path, "a"):
pass
if not max_count:
max_count = 10
+
test_path = get_file_path(file)
- # Probably should enforce only
- # one can be selected via an error on
- # the CLI instead of functionally
- # defining a hierarchy.
+ if not os.path.isfile(test_path):
+
+ if file is not None and len(file) != 0:
+ raise Exception("Specified .toml file not found")
+
+ Path(test_path).touch()
+ print(f"\nCreated {test_path}.")
+ print("Please add your regex and shell tests there.")
+ print("For formatting details see https://github.com/andrewlaack/ratchets\n")
+ exit()
+
+ if not os.path.getsize(test_path):
+ print("No tests defined...")
+ exit()
if blame:
- issues = evaluate_tests(test_path, cmd_mode, regex_mode)
- with_blames = add_blames(issues)
+ issues = evaluate_tests(test_path, cmd_mode, regex_mode, paths)
print_issues_with_blames(issues, max_count)
elif compare_counts:
- issues = evaluate_tests(test_path, cmd_mode, regex_mode)
+ issues = evaluate_tests(test_path, cmd_mode, regex_mode, paths)
current_json = json.loads(results_to_json(issues))
previous_json = load_ratchet_results()
print_diff(current_json, previous_json)
elif update:
- update_ratchets(test_path, cmd_mode, regex_mode)
+ update_ratchets(test_path, cmd_mode, regex_mode, paths)
elif verbose:
- issues = evaluate_tests(test_path, cmd_mode, regex_mode)
+ issues = evaluate_tests(test_path, cmd_mode, regex_mode, paths)
for issue_type in issues:
print_issues(issue_type)
else:
- issues = evaluate_tests(test_path, cmd_mode, regex_mode)
+ issues = evaluate_tests(test_path, cmd_mode, regex_mode, paths)
current_json = json.loads(results_to_json(issues))
print("Current " + str(current_json))
previous_json = load_ratchet_results()
@@ -446,5 +599,7 @@ def cli():
print("Diffs:")
print_diff(current_json, previous_json)
+
if __name__ == "__main__":
+ """Entry point when the file is executed directly, envokes CLI method."""
cli()
diff --git a/src/ratchets/validate.py b/src/ratchets/validate.py
@@ -6,50 +6,58 @@ from .run_tests import (
get_file_path,
)
-def evaluate_single_regex(regex: str, custom_str: str) -> Optional[re.Match[str]]:
+
+def evaluate_single_regex(regex: str, input_str: str) -> Optional[re.Match[str]]:
+ """Evaluate a single regexp based on 'input_str'."""
pattern = re.compile(regex)
- return pattern.search(custom_str)
+ return pattern.search(input_str)
+
-def check_valid(python_tests: Dict[str, Dict[str, Any]]) -> None:
- for test in python_tests:
- regex: str = python_tests[test]["regex"]
- for validation in python_tests[test]["valid"]:
+def check_valid(regex_tests: Dict[str, Dict[str, Any]]) -> None:
+ """Given a dict of regex test and strings, returns if none of the regexps match their strings."""
+ for test in regex_tests:
+ regex: str = regex_tests[test]["regex"]
+ for validation in regex_tests[test]["valid"]:
for line in validation.splitlines():
if evaluate_single_regex(regex, line):
- raise AssertionError(f"Regex: {regex} matched {line}")
+ raise Exception(f"Regex: {regex} matched {line}")
+
-def check_invalid(python_tests: Dict[str, Dict[str, Any]]) -> int:
- for test in python_tests:
- regex: str = python_tests[test]["regex"]
- for validation in python_tests[test]["invalid"]:
+def check_invalid(regex_tests: Dict[str, Dict[str, Any]]) -> int:
+ """Given a dict of regex test and strings, returns if all of the regexps match all of their strings."""
+ for test in regex_tests:
+ regex: str = regex_tests[test]["regex"]
+ for validation in regex_tests[test]["invalid"]:
found: bool = False
for line in validation.splitlines():
if evaluate_single_regex(regex, line):
found = True
if not found:
- raise AssertionError(f"Regex: {regex} not matched in {validation}")
+ raise Exception(f"Regex: {regex} not matched in {validation}")
return 0
-def validate(filename : Optional[str]) -> Optional[bool]:
+
+def validate(filename: Optional[str]) -> Optional[bool]:
+ """Verify the given file's example expressions match the regexps."""
test_path: str = get_file_path(filename)
config: Dict[str, Any] = toml.load(test_path)
- python_tests: Optional[Dict[str, Dict[str, Any]]] = config.get("ratchet", {}).get("regex")
+ regex_tests: Optional[Dict[str, Dict[str, Any]]] = config.get("ratchet", {}).get(
+ "regex"
+ )
- if python_tests is None:
- print("No python tests found, there is nothing to validate.")
+ if regex_tests is None:
+ print("No regex tests found, there is nothing to validate.")
return True
- check_valid(python_tests)
- check_invalid(python_tests)
+ check_valid(regex_tests)
+ check_invalid(regex_tests)
return True
- print(f"All expected regex invalid/valid samples are correct for:\n{test_path}")
-
if __name__ == "__main__":
- parser = argparse.ArgumentParser(description="Python ratchet testing")
+ """Entry point to parse CLI inputs and evaluate .toml test file."""
+ parser = argparse.ArgumentParser(description="Regex ratchet validation")
parser.add_argument("-f", "--file")
args = parser.parse_args()
file: Optional[str] = args.file
validate(file)
-
diff --git a/tests.toml b/tests.toml
@@ -49,4 +49,4 @@ invalid = [
# these will be ran in parallel
[ratchet.shell.line_too_long]
-command = "xargs -n1 awk 'length($0) > 80'"
+command = "xargs -n1 awk 'length($0) > 88'"
diff --git a/tests/file_spec_files/spec_file_1.py b/tests/file_spec_files/spec_file_1.py
@@ -0,0 +1,6 @@
+except:
+except:
+except:
+except:
+except:
+except:
diff --git a/tests/file_spec_files/spec_file_2.py b/tests/file_spec_files/spec_file_2.py
@@ -0,0 +1,8 @@
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
+oairsetnaoristenaorstin aorsient oarisen oarisen oariest oairestnoiares noairest orasitenarsoienarsotienarsotienarsotie
diff --git a/tests/test_files/test_exclusion.py b/tests/test_files/test_exclusion.py
@@ -1,4 +1,3 @@
-
from ratchets import run_tests
from ratchets import abstracted_tests
import os
@@ -14,10 +13,10 @@ def test_config():
assert os.path.isfile(test_path), "tests.toml not found"
try:
- issues = run_tests.evaluate_tests(test_path, True, True)
- run_tests.update_ratchets(test_path, True, True)
+ issues = run_tests.evaluate_tests(test_path, True, True, None)
+ run_tests.update_ratchets(test_path, True, True, None)
except Exception as e:
- assert False, f"Unable to update ratchets using 'tests.toml': {e}"
+ raise Exception(f"Unable to update ratchets using 'tests.toml': {e}")
# TODO:
@@ -25,42 +24,53 @@ def test_config():
# gitignore is handled the same way as
# the excluded file, but should be checked too
+
def test_exclusion():
current_file_directory = os.path.dirname(os.path.abspath(__file__))
- excluded_directory = os.path.abspath(os.path.join(current_file_directory, "..", "excluded_files"))
- test_py_dir= os.path.abspath(os.path.join(current_file_directory, "..", "python_files"))
+ excluded_directory = os.path.abspath(
+ os.path.join(current_file_directory, "..", "excluded_files")
+ )
+ test_py_dir = os.path.abspath(
+ os.path.join(current_file_directory, "..", "python_files")
+ )
exclusion_path = run_tests.get_excludes_path()
root = run_tests.find_project_root()
ignore_path = os.path.join(root, ".gitignore")
-
- python_files_no_exclusion = run_tests.get_python_files(test_py_dir)
+
+ python_files_no_exclusion = run_tests.get_python_files(test_py_dir, None)
# ensure no side effects in the method
- # since we don't change the path values,
- # ensuring the count reamins the same should suffice
+ # since we don't change the path values,
+ # ensuring the count reamins the same should suffice
length_starting = len(python_files_no_exclusion)
-
- expected_results = {
- "default_excluded.txt" : 6,
- "no_1.txt" : 4,
- "no_1_or_dir.txt" : 3
- }
+ expected_results = {"default_excluded.txt": 6, "no_1.txt": 4, "no_1_or_dir.txt": 3}
count = 0
for filename in os.listdir(excluded_directory):
count += 1
full_path = os.path.abspath(os.path.join(excluded_directory, filename))
shutil.copy(full_path, exclusion_path)
- filtered = run_tests.filter_excluded_files(python_files_no_exclusion, exclusion_path, ignore_path)
- assert len(python_files_no_exclusion) == length_starting, "There is a side effect in filter_excluded_files"
-
- assert filename in expected_results, "An additional excluded.txt file was added, but the corresponding expected count was not add to the dict"
- assert expected_results[filename] == len(filtered), "Filter count differs from expected value"
+ filtered = run_tests.filter_excluded_files(
+ python_files_no_exclusion, exclusion_path, ignore_path
+ )
+ if len(python_files_no_exclusion) != length_starting:
+ raise Exception("There is a side effect in filter_excluded_files")
+
+ if not filename in expected_results:
+ raise Exception(
+ "An additional excluded.txt file was added, but the corresponding expected count was not add to the dict"
+ )
+ if not expected_results[filename] == len(filtered):
+ raise Exception("Filter count differs from expected value")
+
+ if count != len(expected_results):
+ raise Exception(
+ "There is an entry in the expected_results dictionary that does not correspond with a file tested"
+ )
- assert count == len(expected_results), "There is an entry in the expected_results dictionary that does not correspond with a file tested"
if __name__ == "__main__":
test_config()
diff --git a/tests/test_files/test_files.py b/tests/test_files/test_files.py
@@ -0,0 +1,53 @@
+from ratchets import run_tests
+from ratchets import abstracted_tests
+import os
+import json
+
+
+def test_files():
+ """Tests the functionallity of TOML and file specification."""
+
+ # directory: Union[str, Path], paths: Optional[List[str]]
+ # ) -> List[Path]:
+
+ proj_root = run_tests.find_project_root()
+
+ file1_path = [os.path.join(proj_root, "tests/file_spec_files/spec_file_1.py")]
+ file2_path = [os.path.join(proj_root, "tests/file_spec_files/spec_file_2.py")]
+
+ filtered1_file = str(abstracted_tests.get_python_files(proj_root, file1_path)[0])
+ filtered2_file = str(abstracted_tests.get_python_files(proj_root, file2_path)[0])
+
+ current_file_directory = os.path.dirname(os.path.abspath(__file__))
+ toml_file_directory = os.path.abspath(
+ os.path.join(current_file_directory, "..", "toml_files")
+ )
+ toml_file = os.path.join(toml_file_directory, "default.toml")
+
+ exceptions1 = run_tests.evaluate_tests(
+ toml_file, False, False, [filtered1_file], True
+ )
+
+ exceptions2 = run_tests.evaluate_tests(
+ toml_file, False, False, [filtered2_file], True
+ )
+
+ json1 = json.loads(run_tests.results_to_json(exceptions1))
+ json2 = json.loads(run_tests.results_to_json(exceptions2))
+
+ exception1_sum = 0
+ for key in json1:
+ exception1_sum += json1[key]
+
+ exception2_sum = 0
+ for key in json2:
+ exception2_sum += json2[key]
+
+ if exception2_sum != 8:
+ raise Exception(f"Incorrect number of infractions counted for {filtered2_file}")
+ if exception1_sum != 6:
+ raise Exception(f"Incorrect number of infractions counted for {filtered1_file}")
+
+if __name__ == "__main__":
+ """Invoke all tests in the file when called directly."""
+ test_files()
diff --git a/tests/test_files/test_toml_configs.py b/tests/test_files/test_toml_configs.py
@@ -6,7 +6,7 @@ from typing import Dict, Any
import json
# verify code still runs as expected even
-# if only cmd or only python sections are defined
+# if only shell or regex sections are defined
# also, ensure sufficiently informative message is shown when
# invalid toml is used.
@@ -15,38 +15,43 @@ import json
# this creates a ratchet_excluded.txt file,
# creates the output json, and verifies there is a default tests.toml file
+
def test_config():
test_path = run_tests.get_file_path(None)
- assert os.path.isfile(test_path), "tests.toml not found"
+ if not os.path.isfile(test_path):
+ raise Exception("tests.toml not found")
try:
- issues = run_tests.evaluate_tests(test_path, True, True)
- run_tests.update_ratchets(test_path, True, True)
+ issues = run_tests.evaluate_tests(test_path, True, True, None)
+ run_tests.update_ratchets(test_path, True, True, None)
except Exception as e:
- assert False, f"Unable to update ratchets using 'tests.toml': {e}"
+ raise Exception(f"Unable to update ratchets using 'tests.toml': {e}")
def test_formatting():
current_file_directory = os.path.dirname(os.path.abspath(__file__))
- toml_file_directory = os.path.abspath(os.path.join(current_file_directory, "..", "toml_files"))
+ toml_file_directory = os.path.abspath(
+ os.path.join(current_file_directory, "..", "toml_files")
+ )
for filename in os.listdir(toml_file_directory):
if filename == "invalid.toml":
try:
full_path = os.path.abspath(os.path.join(toml_file_directory, filename))
- run_tests.evaluate_tests(full_path, True, True)
+ run_tests.evaluate_tests(full_path, True, True, None)
except Exception as e:
- assert isinstance(e, toml.TomlDecodeError), f"Expected TomlDecodeError, got {type(e)}: {e}"
+ if not isinstance(e, toml.TomlDecodeError):
+ raise Exception(f"Expected TomlDecodeError, got {type(e)}: {e}")
else:
- assert False, f"Expected error to be thrown for invalid toml file."
+ raise Exception(f"Expected error to be thrown for invalid toml file.")
else:
full_path = os.path.abspath(os.path.join(toml_file_directory, filename))
-
+
# there is a directory in there
if os.path.isfile(full_path):
- run_tests.evaluate_tests(full_path, True, True)
+ run_tests.evaluate_tests(full_path, True, True, None)
full_path = os.path.abspath(os.path.join(toml_file_directory, filename))
@@ -54,60 +59,71 @@ def test_formatting():
# ensure updated values match subsequent runs.
def verify_updating():
test_path = run_tests.get_file_path(None)
- run_tests.update_ratchets(test_path, True, True)
+ run_tests.update_ratchets(test_path, True, True, None)
# if one is false then the results are guaranteed
# to be either the same or lower.
- issues = run_tests.evaluate_tests(test_path, True, True)
- current_json : Dict [str, Any] = json.loads(run_tests.results_to_json(issues))
- previous_json : Dict[str, Any] = run_tests.load_ratchet_results()
+ issues = run_tests.evaluate_tests(test_path, True, True, None)
+ current_json: Dict[str, Any] = json.loads(run_tests.results_to_json(issues))
+ previous_json: Dict[str, Any] = run_tests.load_ratchet_results()
+
+ if current_json != previous_json:
+ raise Exception(
+ "JSON should be identical when running evals and updating ratchets."
+ )
- assert current_json == previous_json
# test how things behave when ratchet_values.json does not exist
def test_ratchet_excluded_missing():
-
+
ratchet_path = abstracted_tests.get_ratchet_path()
if os.path.isfile(ratchet_path):
try:
os.remove(ratchet_path)
except Exception as e:
- assert False, "Unable to delete ratchet_values.json"
+ raise Exception("Unable to delete ratchet_values.json")
test_path = run_tests.get_file_path(None)
try:
previous = run_tests.load_ratchet_results()
- except Exception as e:
- assert False, "If ratchet_values.json does not exist, we don't throw, assume all 0's"
+ except Exception:
+ raise Exception(
+ "If ratchet_values.json does not exist, we don't throw, assume all 0's"
+ )
- issues = run_tests.evaluate_tests(test_path, True, True)
+ issues = run_tests.evaluate_tests(test_path, True, True, None)
# writes back json file
- run_tests.update_ratchets(test_path, True, True)
+ run_tests.update_ratchets(test_path, True, True, None)
return
+
# test when there are additional values,
# less values, no values (in current).
+
def test_ratchet_values_differ():
-
+
# ensure clean start
test_config()
current_file_directory = os.path.dirname(os.path.abspath(__file__))
- toml_file_directory = os.path.abspath(os.path.join(current_file_directory, "..", "toml_files/different"))
+ toml_file_directory = os.path.abspath(
+ os.path.join(current_file_directory, "..", "toml_files/different")
+ )
for filename in os.listdir(toml_file_directory):
full_path = os.path.abspath(os.path.join(toml_file_directory, filename))
- run_tests.evaluate_tests(full_path, True, True)
+ run_tests.evaluate_tests(full_path, True, True, None)
full_path = os.path.abspath(os.path.join(toml_file_directory, filename))
return
+
if __name__ == "__main__":
test_config()
test_formatting()
diff --git a/tests/test_files/test_validation.py b/tests/test_files/test_validation.py
@@ -1,15 +1,24 @@
from ratchets import validate
import os
+
def test_validate_regex():
current_file_directory = os.path.dirname(os.path.abspath(__file__))
- toml_file_directory_valid = os.path.abspath(os.path.join(current_file_directory, "..", "toml_files/regexp/valid"))
- toml_file_directory_invalid = os.path.abspath(os.path.join(current_file_directory, "..", "toml_files/regexp/invalid"))
+ toml_file_directory_valid = os.path.abspath(
+ os.path.join(current_file_directory, "..", "toml_files/regexp/valid")
+ )
+ toml_file_directory_invalid = os.path.abspath(
+ os.path.join(current_file_directory, "..", "toml_files/regexp/invalid")
+ )
for filename in os.listdir(toml_file_directory_valid):
full_path = os.path.abspath(os.path.join(toml_file_directory_valid, filename))
if os.path.isfile(full_path):
- assert validate.validate(full_path), f"{full_path}, was deemed to be invalid"
+ try:
+ # Throws if not valid
+ validate.validate(full_path)
+ except Exception as e:
+ raise Exception(f"{full_path}, was deemed to be invalid \n {e}")
for filename in os.listdir(toml_file_directory_invalid):
full_path = os.path.abspath(os.path.join(toml_file_directory_invalid, filename))
@@ -20,7 +29,10 @@ def test_validate_regex():
except Exception:
pass
else:
- assert False, f"Expected validation to fail for {full_path}, but it passed"
+ raise Exception(
+ f"Expected validation to fail for {full_path}, but it passed"
+ )
+
if __name__ == "__main__":
test_validate_regex()
diff --git a/tests/toml_files/default.toml b/tests/toml_files/default.toml
@@ -49,4 +49,4 @@ invalid = [
# these will be ran in parallel
[ratchet.shell.line_too_long]
-command = "xargs -n1 awk 'length($0) > 80'"
+command = "xargs -n1 awk 'length($0) > 88'"
