From 1f5c6ba102de7faae72b6d89d03841e910c4e234 Mon Sep 17 00:00:00 2001 From: hightemp Date: Thu, 22 Oct 2026 18:13:13 +0300 Subject: [PATCH 1/3] docs: plan the URL/NUL normalizer (TASKS 1.4) Co-Authored-By: Claude Opus 5.5 --- PRD.md | 30 ++++++++++++++++++++++++++++++ PRD.ru.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/PRD.md b/PRD.md index 798c733..a12222c 100644 --- a/PRD.md +++ b/PRD.md @@ -609,3 +609,33 @@ optional Redis/Memcached. - lookup by bit: found, 0, several bits, unknown bit; - `ig_detector_at` out of range; - registry `detect` gives the same verdicts as the direct functions. + +### 1.4 Normalizer: URL decoding, `%u`, NUL +- **Context:** + - PHP has already URL-decoded GET/POST/COOKIE once before the input filter, so `decode_rounds` (default 2) are **extra** rounds that catch double and triple encoding. + - `REQUEST_URI` is raw, so the same rounds give it its first decode. + - A copy is normalized; the original value is not touched. +- **Decision** (`src/core/ig_normalize.{h,c}`): + - **One round:** `ig_url_decode(in, len, out, flags)`. + - `%XX` → byte. + - `%uXXXX` (IIS, a common WAF bypass) → UTF-8, 1–3 bytes; surrogates are not decoded. + - `+` → space only with `IG_DECODE_PLUS` (query/body; it is literal in a URI path). + - Invalid or truncated sequences are copied as is, like PHP does. + - The output is never longer than the input, so an `out` of `len` bytes is enough and in-place decoding is allowed. + - **Normalization:** `ig_normalize(in, len, rounds, flags, out, &stats)`. + - Each round decodes, then removes NUL bytes. NUL is removed in every round because `%%0041` becomes `%41` only after the NUL goes. + - It stops when a round changes nothing or the rounds run out. + - NUL removal also happens with `rounds = 0`. + - **Stats:** `rounds` (rounds that changed the value), `nul_removed`, and `limit_hit` (the value would still decode). The anomaly event for `limit_hit` comes with class 3 (v1.1). + - **No allocation** in the core: the caller provides the buffer (`emalloc` in the Zend layer, phase 4). Stateless and ZTS-safe. + - **Out of scope:** overlong UTF-8 and invalid encodings (class 3), HTML entities (1.5), length limits (1.6). +- **Tests first** (`tests/unit/test_normalize.c`): + - one round: + - hex in both cases; invalid and truncated sequences kept; + - `%u0041`, `%u0410`, `%u20AC`; a surrogate and an invalid `%u` kept; + - `+` with and without the flag; in-place decoding; empty input. + - normalization: + - double encoding; triple encoding with 2 rounds (→ `limit_hit`); early stop; + - NUL raw and `%00`; `%%0041`; + - `rounds = 0`; the output is never longer; NULL arguments. + - Fuzz target `fuzz_normalize`: output ≤ input and NUL-free, in-place = separate buffer, a round never grows the value. diff --git a/PRD.ru.md b/PRD.ru.md index 15c4f70..3361ee9 100644 --- a/PRD.ru.md +++ b/PRD.ru.md @@ -1128,3 +1128,48 @@ SEVERITY = "info" | "low" | "medium" | "high" | "critical" ; - поиск по биту: найден, 0, несколько битов, неизвестный бит; - `ig_detector_at` вне диапазона; - `detect` из реестра — та же функция, что `ig_detect_sqli`/`ig_detect_xss`: тот же вердикт на атаке. + +### 1.4 Нормализатор: URL-decode, `%u`, NUL +**Контекст:** +- GET, POST и COOKIE PHP уже один раз раскодировал (`urldecode`) до input filter. Поэтому `decode_rounds` (по умолчанию 2) — это **дополнительные** раунды: они ловят двойное и тройное кодирование (`%2527` → `%27` → `'`). +- URI (`REQUEST_URI`) приходит сырым, для него те же раунды дают первое раскодирование. +- Нормализуется копия: исходное значение не меняется (FR-1.3). + +**Решение (`src/core/ig_normalize.{h,c}`):** +- **Один раунд:** `size_t ig_url_decode(const char *in, size_t len, char *out, unsigned flags)`. + - `%XX` (hex в любом регистре) → байт. + - `%uXXXX` (IIS/старые IE, частый приём обхода WAF) → кодовая точка в UTF-8, 1–3 байта; суррогаты `D800–DFFF` не раскодируются. + - `+` → пробел только с флагом `IG_DECODE_PLUS`: как `urldecode()` для query и тела. Для пути URI `+` — литерал. + - Невалидные и обрезанные последовательности (`%zz`, `%4` в конце, `%u12`) копируются как есть, ровно как в PHP. + - Результат никогда не длиннее входа: `%XX` даёт 1 байт из 3, `%uXXXX` — не более 3 из 6. Буфер `out` размером `len` достаточен, `out == in` (на месте) допустим. +- **Нормализация:** `ig_normalize(in, len, rounds, flags, out, &stats)` → длина результата. + - Раунд = URL-decode, затем удаление NUL-байтов. + - NUL удаляется в каждом раунде, а не только в конце: иначе `%%0041` → `%`+NUL+`41` → после удаления `%41` — новая последовательность, которую должен увидеть следующий раунд. + - Остановка — когда раунд ничего не изменил или раунды кончились. + - NUL удаляется и при `rounds = 0`. +- **Статистика `ig_norm_stats_t`:** + - `rounds` — сколько раундов изменили значение; + - `nul_removed` — сколько удалено NUL; + - `limit_hit` — после последнего разрешённого раунда значение всё ещё раскодируется. + + Глубокое кодирование само по себе подозрительно; событие аномалии будет в классе 3 (v1.1), сейчас флаг только возвращается. +- **Без выделения памяти** в ядре: буфер даёт вызывающий (Zend-слой — `emalloc`, фаза 4). Состояния нет, функции безопасны для ZTS. +- **Вне задачи:** + - overlong UTF-8 (`%c0%af`) и невалидные кодировки — класс 3 (аномалии ввода); + - HTML-сущности — 1.5; + - ограничение длины — 1.6. + +**Тесты (red первыми), `tests/unit/test_normalize.c`:** +- Один раунд: + - `%XX` в верхнем и нижнем регистре; невалидные и обрезанные последовательности сохраняются; + - `%u0041` → `A`, `%u0410` → `А` (UTF-8), `%u20AC` → `€`; суррогат и невалидный `%u` сохраняются; + - `+` с флагом и без; раскодирование на месте; пустой вход. +- Нормализация: + - `%2527` при 2 раундах → `'`; тройное кодирование при 2 раундах → `%27` и `limit_hit`; + - ранняя остановка (`rounds` в статистике); + - удаление NUL: сырой байт и `%00`; случай `%%0041`; + - `rounds = 0`; результат не длиннее входа; NULL-аргументы. +- Fuzz-таргет `fuzz_normalize`: + - результат ≤ входа и без NUL; + - на месте = в отдельный буфер; + - один раунд никогда не увеличивает длину. From a8f3f777f23a6e758bc41df1e5c2a4a798306ee6 Mon Sep 17 00:00:00 2001 From: hightemp Date: Thu, 22 Oct 2026 18:13:56 +0300 Subject: [PATCH 2/3] test: URL/NUL normalizer (red) Co-Authored-By: Claude Opus 5.5 --- tests/fuzz/corpus/fuzz_normalize/double | 1 + tests/fuzz/corpus/fuzz_normalize/nul | 1 + tests/fuzz/corpus/fuzz_normalize/percent_u | Bin 0 -> 19 bytes tests/fuzz/fuzz_normalize.c | 54 ++++++ tests/unit/test_normalize.c | 192 +++++++++++++++++++++ 5 files changed, 248 insertions(+) create mode 100644 tests/fuzz/corpus/fuzz_normalize/double create mode 100644 tests/fuzz/corpus/fuzz_normalize/nul create mode 100644 tests/fuzz/corpus/fuzz_normalize/percent_u create mode 100644 tests/fuzz/fuzz_normalize.c create mode 100644 tests/unit/test_normalize.c diff --git a/tests/fuzz/corpus/fuzz_normalize/double b/tests/fuzz/corpus/fuzz_normalize/double new file mode 100644 index 0000000..9cdf7a6 --- /dev/null +++ b/tests/fuzz/corpus/fuzz_normalize/double @@ -0,0 +1 @@ +%2527%2520OR%25201%253D1 \ No newline at end of file diff --git a/tests/fuzz/corpus/fuzz_normalize/nul b/tests/fuzz/corpus/fuzz_normalize/nul new file mode 100644 index 0000000..26e6972 --- /dev/null +++ b/tests/fuzz/corpus/fuzz_normalize/nul @@ -0,0 +1 @@ +%%0041+x \ No newline at end of file diff --git a/tests/fuzz/corpus/fuzz_normalize/percent_u b/tests/fuzz/corpus/fuzz_normalize/percent_u new file mode 100644 index 0000000000000000000000000000000000000000..9bd29b8dd97b16c85d0b451d748e76b16563dabc GIT binary patch literal 19 acmZQjEj2JPG*B%yGH`TOEp@RlFaQ84PXtx~ literal 0 HcmV?d00001 diff --git a/tests/fuzz/fuzz_normalize.c b/tests/fuzz/fuzz_normalize.c new file mode 100644 index 0000000..cb428e9 --- /dev/null +++ b/tests/fuzz/fuzz_normalize.c @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: MIT */ +/* Fuzz target: the normalizer never grows a value, leaves no NUL and gives + * the same result in place and into a separate buffer. */ + +#include "core/ig_normalize.h" + +#include +#include +#include + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + char *out; + char *inplace; + size_t n; + size_t m; + unsigned int flags; + ig_norm_stats_t st; + + if (size == 0) { + return 0; + } + /* the first byte selects the options */ + flags = (data[0] & 1u) ? IG_DECODE_PLUS : 0; + data++; + size--; + + out = malloc(size + 1); + inplace = malloc(size + 1); + if (out == NULL || inplace == NULL) { + abort(); + } + + n = ig_url_decode((const char *)data, size, out, flags); + if (n > size) { + abort(); + } + + n = ig_normalize((const char *)data, size, 3, flags, out, &st); + if (n > size || memchr(out, '\0', n) != NULL || st.rounds > 3) { + abort(); + } + memcpy(inplace, data, size); + m = ig_normalize(inplace, size, 3, flags, inplace, NULL); + if (m != n || memcmp(out, inplace, n) != 0) { + abort(); + } + + free(out); + free(inplace); + return 0; +} diff --git a/tests/unit/test_normalize.c b/tests/unit/test_normalize.c new file mode 100644 index 0000000..66f78db --- /dev/null +++ b/tests/unit/test_normalize.c @@ -0,0 +1,192 @@ +/* SPDX-License-Identifier: MIT */ +/* Normalizer: repeated URL decoding, %uXXXX, NUL removal (FR-1.3). */ + +#include "ig_test.h" +#include "core/ig_normalize.h" + +#include + +/* decode1 : one round into a separate buffer */ +static int decode1(const char *in, unsigned int flags, const char *want) +{ + char out[128]; + size_t n = ig_url_decode(in, strlen(in), out, flags); + + if (n != strlen(want) || memcmp(out, want, n) != 0) { + ig_test_fail(__FILE__, __LINE__, "decode \"%s\": got \"%.*s\" (%zu), expected \"%s\"", in, + (int)(n < sizeof(out) ? n : 0), out, n, want); + return 0; + } + return 1; +} + +/* norm : full normalization */ +static int norm( + const char *in, size_t len, unsigned int rounds, const char *want, ig_norm_stats_t *st) +{ + char out[128]; + size_t n = ig_normalize(in, len, rounds, IG_DECODE_PLUS, out, st); + + if (n != strlen(want) || memcmp(out, want, n) != 0) { + ig_test_fail(__FILE__, __LINE__, "normalize \"%s\": got \"%.*s\" (%zu), expected \"%s\"", + in, (int)(n < sizeof(out) ? n : 0), out, n, want); + return 0; + } + return 1; +} + +IG_TEST(url_decode_hex_in_both_cases) +{ + if (!decode1("%27%20OR%201%3d1", 0, "' OR 1=1")) { + return; + } + if (!decode1("%3C%3c%3E%3e", 0, "<<>>")) { + return; + } + decode1("plain", 0, "plain"); +} + +IG_TEST(url_decode_keeps_invalid_and_truncated_sequences) +{ + if (!decode1("%zz%g1%1g", 0, "%zz%g1%1g")) { + return; + } + if (!decode1("100%", 0, "100%")) { + return; + } + if (!decode1("a%4", 0, "a%4")) { + return; + } + decode1("%%41", 0, "%A"); +} + +IG_TEST(url_decode_percent_u) +{ + if (!decode1("%u0041", 0, "A")) { + return; + } + if (!decode1("%u0410%U0411", 0, "\xd0\x90\xd0\x91")) { /* "АБ" */ + return; + } + if (!decode1("%u20AC", 0, "\xe2\x82\xac")) { /* "€" */ + return; + } + if (!decode1("%u0027%u003c", 0, "'<")) { + return; + } + /* surrogates, invalid hex and truncated forms are kept */ + if (!decode1("%uD800%udfff", 0, "%uD800%udfff")) { + return; + } + if (!decode1("%u12g4", 0, "%u12g4")) { + return; + } + decode1("%u123", 0, "%u123"); +} + +IG_TEST(url_decode_plus_only_with_flag) +{ + if (!decode1("1+OR+1", IG_DECODE_PLUS, "1 OR 1")) { + return; + } + decode1("1+OR+1", 0, "1+OR+1"); +} + +IG_TEST(url_decode_in_place_and_empty) +{ + char buf[] = "%27%u0041+x"; + size_t n = ig_url_decode(buf, strlen(buf), buf, IG_DECODE_PLUS); + + IG_ASSERT_INT_EQ(4, n); + IG_ASSERT_TRUE(memcmp(buf, "'A x", 4) == 0); + IG_ASSERT_INT_EQ(0, ig_url_decode("", 0, buf, 0)); +} + +IG_TEST(normalize_decodes_double_encoding) +{ + ig_norm_stats_t st; + + if (!norm("%2527%2520OR%25201%253D1", 24, 2, "' OR 1=1", &st)) { + return; + } + IG_ASSERT_INT_EQ(2, st.rounds); + IG_ASSERT_FALSE(st.limit_hit); +} + +IG_TEST(normalize_reports_the_round_limit) +{ + ig_norm_stats_t st; + + /* triple encoding with two rounds stops at %27 */ + if (!norm("%252527", 7, 2, "%27", &st)) { + return; + } + IG_ASSERT_INT_EQ(2, st.rounds); + IG_ASSERT_TRUE(st.limit_hit); +} + +IG_TEST(normalize_stops_when_stable) +{ + ig_norm_stats_t st; + + if (!norm("%27x", 4, 5, "'x", &st)) { + return; + } + IG_ASSERT_INT_EQ(1, st.rounds); + IG_ASSERT_FALSE(st.limit_hit); + + if (!norm("plain text", 10, 5, "plain text", &st)) { + return; + } + IG_ASSERT_INT_EQ(0, st.rounds); +} + +IG_TEST(normalize_removes_nul_bytes) +{ + ig_norm_stats_t st; + + if (!norm("a\0b%00c", 7, 2, "abc", &st)) { + return; + } + IG_ASSERT_INT_EQ(2, st.nul_removed); + + /* the NUL hides a sequence until it is removed: %41 -> %41 -> A */ + if (!norm("%%0041", 6, 2, "A", &st)) { + return; + } + IG_ASSERT_INT_EQ(1, st.nul_removed); +} + +IG_TEST(normalize_with_zero_rounds_only_removes_nul) +{ + ig_norm_stats_t st; + + if (!norm("%27\0x", 5, 0, "%27x", &st)) { + return; + } + IG_ASSERT_INT_EQ(0, st.rounds); + IG_ASSERT_INT_EQ(1, st.nul_removed); + IG_ASSERT_TRUE(st.limit_hit); +} + +IG_TEST(normalize_in_place_and_without_stats) +{ + char buf[] = "%2527x"; + size_t n = ig_normalize(buf, 6, 2, 0, buf, NULL); + + IG_ASSERT_INT_EQ(2, n); + IG_ASSERT_TRUE(memcmp(buf, "'x", 2) == 0); +} + +IG_TEST(normalize_null_arguments) +{ + char out[8]; + ig_norm_stats_t st; + + IG_ASSERT_TRUE(ig_normalize(NULL, 3, 2, 0, out, &st) == IG_NORM_ERROR); + IG_ASSERT_TRUE(ig_normalize("abc", 3, 2, 0, NULL, &st) == IG_NORM_ERROR); + IG_ASSERT_INT_EQ(0, ig_normalize(NULL, 0, 2, 0, out, &st)); + IG_ASSERT_INT_EQ(0, st.rounds); + IG_ASSERT_TRUE(ig_url_decode(NULL, 3, out, 0) == IG_NORM_ERROR); + IG_ASSERT_TRUE(ig_url_decode("abc", 3, NULL, 0) == IG_NORM_ERROR); +} From 4a17f77b13d41bf248c32c2b2e89327840197f14 Mon Sep 17 00:00:00 2001 From: hightemp Date: Thu, 22 Oct 2026 18:17:11 +0300 Subject: [PATCH 3/3] core: URL/NUL normalizer (green) Co-Authored-By: Claude Opus 5.5 --- TASKS.md | 2 +- TASKS.ru.md | 2 +- config.m4 | 2 +- src/core/ig_normalize.c | 167 ++++++++++++++++++++++++++++++++++++++++ src/core/ig_normalize.h | 33 ++++++++ 5 files changed, 203 insertions(+), 3 deletions(-) create mode 100644 src/core/ig_normalize.c create mode 100644 src/core/ig_normalize.h diff --git a/TASKS.md b/TASKS.md index 89970a6..c503e7a 100644 --- a/TASKS.md +++ b/TASKS.md @@ -53,7 +53,7 @@ - [x] **1.1 [TDD]** `ig_detect_sqli()` wrapper over libinjection, fingerprint in result. - [x] **1.2 [TDD]** `ig_detect_xss()` wrapper. - [x] **1.3 [TDD]** Common detector interface and registry. -- [ ] **1.4 [TDD]** Normalizer: repeated URL decoding, NUL removal, `%u` sequences. +- [x] **1.4 [TDD]** Normalizer: repeated URL decoding, NUL removal, `%u` sequences. - [ ] **1.5 [TDD]** HTML entity decoding for XSS. - [ ] **1.6 [TDD]** Length limit with `truncated` flag. - [ ] **1.7 [TDD]** Glob matcher for exclusions. diff --git a/TASKS.ru.md b/TASKS.ru.md index 7ba7af8..489999d 100644 --- a/TASKS.ru.md +++ b/TASKS.ru.md @@ -54,7 +54,7 @@ - [x] **1.1 [TDD]** Обёртка `ig_detect_sqli(const char*, size_t, ig_result_t*)` над libinjection: тесты на классические payload'ы (`' OR 1=1--`, `UNION SELECT`, `; DROP TABLE`) и на безопасные строки. Fingerprint в результате. (FR-2.1) - [x] **1.2 [TDD]** Обёртка `ig_detect_xss(...)`: `