diff --git a/README.md b/README.md deleted file mode 100644 index b72d961..0000000 --- a/README.md +++ /dev/null @@ -1,57 +0,0 @@ -![Kapow!](https://trello-attachments.s3.amazonaws.com/5c6edee98297dc18aa4e2b63/960x720/ff8d28fc24af11e3295afa5a9665bdc0/kapow-1601675_960_720.png) - -**Kapow!** allows you to leverage the Ultimate Power™ of the UNIX® shell via HTTP. - -**Kapow!** is in the process of being defined by a [specification](/spec/) - -# Sample usage -## Clone the project -```bash -# clone this project -``` - -## Build the kapow! docker image -```bash -docker build -t bbva/kapow:0.1 /path/to/kapow/poc -``` - -## Build a docker image for running the nmap example -```bash -docker build -t kapow-nmap /path/to/kapow/poc/examples/nmap -``` - -## Run kapow -```bash -docker run \ - -it \ - -p 8080:8080 \ - kapow-nmap -``` -which will output something like this: -``` -======== Running on http://0.0.0.0:8080 ======== -(Press CTRL+C to quit) -Route created POST /list/{ip} -ROUTE_8ed01c48_bf23_455a_8186_a1df7ab09e48 -bash-4.4# -``` - - -## Test /list endpoint -In another terminal, try running: -```bash -curl http://localhost:8080/list/github.com -``` -which will respond something like: -``` -Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-10 14:01 UTC -Nmap scan report for github.com (140.82.118.3) -rDNS record for 140.82.118.3: lb-140-82-118-3-ams.github.com -Nmap done: 1 IP address (0 hosts up) scanned in 0.04 seconds - -``` -et voilà ! - -# License - -This project is distributed under the [Apache License 2.0](/LICENSE). diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..c95cbfe --- /dev/null +++ b/README.rst @@ -0,0 +1,82 @@ +.. image:: https://trello-attachments.s3.amazonaws.com/5c824318411d973812cbef67/5ca1af818bc9b53e31696de3/f51eb40412bf09c8c800511d7bbe5634/kapow-1601675_480.png + :alt: Kapow! + +**Kapow!** If you can script it, you can HTTP it. + ++-----------------+------------------------------------------------+ +| Project site | https://github.com/BBVA/kapow | ++-----------------+------------------------------------------------+ +| Issues | https://github.com/BBVA/kapow/issues | ++-----------------+------------------------------------------------+ +| Specification | https://github.com/BBVA/kapow/tree/master/spec | ++-----------------+------------------------------------------------+ +| Documentation | https://github.com/BBVA/kapow/tree/master/doc | ++-----------------+------------------------------------------------+ +| Python versions | 3.7 or above | ++-----------------+------------------------------------------------+ + + +CAVEAT EMPTOR +============= + +**Warning!!! Kapow!** is under **heavy development** and [specification](/spec/); +the provided code is a Proof of Concept and the final version will not even +share programming language. Ye be warned. + + +How was born +------------ + +Some awesome history is coming. + + +Kapow! for the impatient +======================== + +When you need to **share** a ``command`` but **not** a complete remote ``ssh +access``, Kapow! will help you by the power of HTTP: + +.. image:: https://trello-attachments.s3.amazonaws.com/5c824318411d973812cbef67/5ca1af818bc9b53e31696de3/784a183fba3f24872dd97ee28e765922/Kapow!.png + :alt: Where Kapow! lives + +Kapow! allows you to write a litte script that will **serve an executable as REST +service**. This script will let you define how to connect HTTP and the Shell +using Kapow!'s shell abstractions to the HTTP world. See it to believe: + +.. image:: https://github.com/BBVA/kapow/blob/develop/resources/kapow.gif?raw=true + :alt: Kapow! in action + + +Superpowers +----------- + +Kapow! gives you: + + * A very simple way to turn any shell **executable into an API** + * A **remote administration** API + * A way to define the integration in you own terms, obligations-free! + + +Curses +------ + +Kapow! can't help when: + + * You need high throughput: Kapow! spawns a new executable for every HTTP call + * You must perform complex logic to attend the request: never use Kapow! if + your executables don't perform al least 90% of the hard work + * You are building a huge application + + +When is your best ally: +----------------------- + + * Easy command + Hard API = Kapow! to the rescue + * SSH for one command? Kapow! allows you to share only that command + * Remote instrumentation of several machines? make it easy with Kapow! + + +The more you know +================= + +If you want to know more, please follow our `documentation `_. diff --git a/doc/readme.rst b/doc/readme.rst new file mode 100644 index 0000000..6f47f80 --- /dev/null +++ b/doc/readme.rst @@ -0,0 +1,189 @@ +What is Kapow! +============== + +Kapow! is an adapter between the world of Pure UNIX® Shell and an HTTP service. + +Some tasks are more convenient in the shell, like cloud interactions, or some +administrative tools. On the other hand, some tasks are more convenient as a +service, like DevSecOps tooling. + +Kapow! lies between these two worlds, making your life easier. Maybe you wonder +about how this kind of magic can happen; if you want to know the nitty-gritty +details, just read our [spec](/spec/). Or, if you want to know how Kapow! can +help you first, let's start with a common situation. + +Think about that awesome command that you use every day, something very +familiar, like ``cloudx storage ls /backups``. Then someone asks you for an +specific backup, so you ``ssh`` into the host, execute your command, possibly +``grepping`` through its output, copy the result and send it back to him. +And that's fine... for the 100 first times. + +Then you decide, let's use an API for this and generate an awesome web server +with it. So, you create a project, manage its dependencies, code the server, +parse the request, learn how to use the API, call the API and deploy it +somewhere. And that's fine... until you find yourself again in the same +situation with another awesome command. + +The awesomeness of UNIX® commands is infinite, so you'll be in this situation +an infinite number of times! Instead, let's put Kapow! into action. + +With Kapow!, when someone asks you for an specific backup (remember your +familiar command?) you just need to create a ``.pow`` file named ``backups.pow`` +that contains: + +.. code-block:: bash + + kapow route add /backups \ + -c 'cloudx storage ls /backups | grep $(request /params/query) | response /body' + +and execute it in the host with the command: + +.. code-block:: bash + + kapow server backups.pow + +and that's it. Done. Do you like it? yes? Then let's start learning a little +more. + + +The mandatory Hello World (for WWW fans) +---------------------------------------- + +First you must create a pow file named ``hello.pow`` with the following contents: + +.. code-block:: bash + + kapow route add /greet -c "echo 'hello world' | response /body" + +then, you must execute: + +.. code-block:: bash + + kapow server hello.pow + +and you can check that it works as intended with good ole' ``curl``: + +.. code-block:: bash + + curl localhost:8080/greet + + +The mandatory Echo (for UNIX fans) +---------------------------------- + +First you must create a pow file named ``echo.pow`` with the following contents: + +.. code-block:: bash + + kapow route add -X POST /echo -c 'request /body | response /body' + +then, you must execute: + +.. code-block:: bash + + kapow server echo.pow + +and you can check that it works as intended with good ole ``curl``: + +.. code-block:: bash + + curl -X POST -d '1,2,3... testing' localhost:8080/echo + + +The multiline fun +----------------- + +Unless you're a hardcore Perl hacker, you'll probably need to write your stuff +over more than one line. + +Don't worry, we need to write several lines, too. Bash, in its magnificent +UNIX® style, provides us with the +`here-documents`_ mechanism that we can leverage precisely for this purpose. + +.. _here-documents: https://www.gnu.org/software/bash/manual/bash.html#Here-Documents + +Let's write a ``multiline.pow`` file with the following content: + +.. code-block:: bash + + kapow route add /log_and_love - <<- 'EOF' + echo "[$(date)] and stuff" >> stuff.log + echo love | response /body + EOF + +and then we serve it with ``kapow``: + +.. code-block:: bash + + kapow server multiline.pow + +Yup. As simple as that. + + +Sample Docker usage +=================== + +Clone the project +----------------- + +.. code-block:: bash + + # clone this project + + +Build the kapow! docker image +----------------------------- + +.. code-block:: bash + + docker build -t bbva/kapow:0.1 /path/to/kapow/poc + +Build a docker image for running the nmap example + +.. code-block:: bash + + docker build -t kapow-nmap /path/to/kapow/poc/examples/nmap + +Run kapow +--------- +.. code-block:: bash + + docker run \ + -it \ + -p 8080:8080 \ + kapow-nmap + +which will output something like this: + +.. code-block:: bash + + ======== Running on http://0.0.0.0:8080 ======== + (Press CTRL+C to quit) + Route created POST /list/{ip} + ROUTE_8ed01c48_bf23_455a_8186_a1df7ab09e48 + bash-4.4# + + +Test /list endpoint +------------------- +In another terminal, try running: + +.. code-block:: bash + + curl http://localhost:8080/list/github.com + +which will respond something like: + +.. code-block:: bash + + Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-10 14:01 UTC + Nmap scan report for github.com (140.82.118.3) + rDNS record for 140.82.118.3: lb-140-82-118-3-ams.github.com + Nmap done: 1 IP address (0 hosts up) scanned in 0.04 seconds + +et voilà ! + +License +======= + +This project is distributed under the [Apache License 2.0](/LICENSE). diff --git a/poc/Dockerfile b/poc/Dockerfile index b4e5d78..79a15b5 100644 --- a/poc/Dockerfile +++ b/poc/Dockerfile @@ -1,8 +1,20 @@ FROM python:3.7-alpine -RUN apk update && apk add bash curl coreutils file -RUN pip install pipenv + COPY Pipfile Pipfile.lock /tmp/ -RUN cd /tmp && pipenv install --system --deploy -COPY kapow /usr/bin + COPY bin/* /usr/bin/ + +WORKDIR /tmp + +RUN apk upgrade --update-cache; \ + apk add \ + bash \ + curl \ + coreutils \ + file; \ + \ + pip install pipenv; \ + \ + pipenv install --system --deploy; + ENTRYPOINT ["/usr/bin/kapow"] diff --git a/poc/kapow b/poc/bin/kapow similarity index 75% rename from poc/kapow rename to poc/bin/kapow index c2183c0..621fd9d 100755 --- a/poc/kapow +++ b/poc/bin/kapow @@ -26,6 +26,9 @@ import shlex import sys from aiohttp import web, StreamReader +import click +import requests + log = logging.getLogger('kapow') @@ -69,17 +72,19 @@ class Connection: return self.request.content elif res.path == 'request/path': return self.request.path.encode('utf-8') - elif res.path.startswith('request/match/'): + elif res.path == 'request/host': + return self.request.host.encode('utf-8') + elif res.path.startswith('request/matches/'): return self.request.match_info[nth(2)].encode('utf-8') - elif res.path.startswith('request/param/'): + elif res.path.startswith('request/params/'): return self.request.rel_url.query[nth(2)].encode('utf-8') - elif res.path.startswith('request/header/'): + elif res.path.startswith('request/headers/'): return self.request.headers[nth(2)].encode('utf-8') - elif res.path.startswith('request/cookie/'): + elif res.path.startswith('request/cookies/'): return self.request.cookies[nth(2)].encode('utf-8') elif res.path.startswith('request/form/'): return (await self.request.post())[nth(2)].encode('utf-8') - elif res.path.startswith('request/file/'): + elif res.path.startswith('request/files/'): name = nth(2) content = nth(3) # filename / content field = (await self.request.post())[name] @@ -109,10 +114,10 @@ class Connection: self._status = int((await content.read()).decode('utf-8')) elif res.path == 'response/body': self._body.write(await content.read()) - elif res.path.startswith('response/header/'): + elif res.path.startswith('response/headers/'): clean = (await content.read()).rstrip(b'\n').decode('utf-8') self._headers[nth(2)] = clean - elif res.path.startswith('response/cookie/'): + elif res.path.startswith('response/cookies/'): clean = (await content.read()).rstrip(b'\n').decode('utf-8') self._cookies[nth(2)] = clean elif res.path == 'response/stream': @@ -227,8 +232,8 @@ def handle_route(entrypoint, command): shell_task = await asyncio.create_subprocess_shell( args, env={**os.environ, - "KAPOW_URL": "http://localhost:8080/kapow", - "KAPOW_CONNECTION": id + "KAPOW_URL": "http://localhost:8080", + "KAPOW_HANDLER_ID": id }, stdin=asyncio.subprocess.DEVNULL) @@ -253,7 +258,7 @@ async def get_routes(request): return web.json_response(list(request.app.router)) -async def create_route(request): +async def append_route(request): """Create a new Kapow! route.""" request.app.router._frozen = False content = await request.json() @@ -281,24 +286,23 @@ async def delete_route(request): ######################################################################## -async def run_init_script(app): +async def run_init_script(app, scripts): """ Run the init script if given, then wait for the shell to finish. """ - if len(sys.argv) == 1: + if not scripts: # No script given cmd = "/bin/bash" - elif len(sys.argv) == 2: - cmd = f"/bin/bash --init-file {sys.argv[1]}" else: - print(f"Usage: {sys.argv[0]} ") - os._exit(1) + filenames = " ".join(shlex.quote(f) for f in scripts) + cmd = f"/bin/bash --init-file <(cat {filenames})" shell_task = await asyncio.create_subprocess_shell( cmd, + executable="/bin/bash", env={**os.environ, - "KAPOW_URL": "http://localhost:8080/kapow" + "KAPOW_URL": "http://localhost:8080" }) await shell_task.wait() @@ -308,23 +312,88 @@ async def run_init_script(app): async def start_background_tasks(app): loop = asyncio.get_running_loop() - app["debug_tasks"] = loop.create_task(run_init_script(app)) + app["debug_tasks"] = loop.create_task(run_init_script(app, app["scripts"])) -def kapow(): - """Start aiohttp app.""" +def start_kapow_server(scripts): app = web.Application(client_max_size=1024**3) app.add_routes([ - web.get('/kapow/routes', get_routes), - web.post('/kapow/routes', create_route), - web.delete('/kapow/routes/{id}', delete_route), - web.get('/kapow/connections/{id}/{field:.*}', get_field), - # web.post('/kapow/connections/{id}/{field:.*}', append_field), - web.put('/kapow/connections/{id}/{field:.*}', set_field), + # Control API + web.get('/routes', get_routes), + web.post('/routes', append_route), # TODO: return route index + # web.put('/routes', insert_route), # TODO: return route index + web.delete('/routes/{id}', delete_route), + + # Data API + web.get('/handlers/{id}/{field:.*}', get_field), + web.put('/handlers/{id}/{field:.*}', set_field), ]) + app["scripts"] = scripts app.on_startup.append(start_background_tasks) web.run_app(app) +######################################################################## +# Command Line # +######################################################################## + + +@click.group() +@click.pass_context +def kapow(ctx): + """Start aiohttp app.""" + pass + + +@kapow.command() +@click.argument("scripts", nargs=-1) +def server(scripts): + start_kapow_server(scripts) + +@kapow.group() +def route(): + pass + + +@route.command() +@click.option("-c", "--command", nargs=1) +@click.option("-e", "--entrypoint", default="/bin/sh -c") +@click.option("-X", "--method", default="GET") +@click.option("--url", envvar='KAPOW_URL') +@click.argument("url_pattern", nargs=1) +@click.argument("command_file", required=False) +def add(url_pattern, entrypoint, command, method, url, command_file): + if command: + # Command is given inline + source = command + elif command_file is None: + # No command + source = "" + elif command_file == '-': + # Read commands from stdin + source = sys.stdin.read() + else: + # Read commands from a file + with open(command_file, 'r', encoding='utf-8') as handler: + source = handler.read() + + response = requests.post(f"{url}/routes", + json={"method": method, + "url_pattern": url_pattern, + "entrypoint": entrypoint, + "command": source}) + response.raise_for_status() + print(response.json()) + + +@route.command() +@click.option("--url", envvar='KAPOW_URL') +@click.argument("route-id") +def remove(route_id, url): + response = requests.delete(f"{url}/routes/{route_id}") + response.raise_for_status() + print(response.json()) + + if __name__ == '__main__': kapow() diff --git a/poc/bin/kroute b/poc/bin/kroute deleted file mode 100755 index f60e9e1..0000000 --- a/poc/bin/kroute +++ /dev/null @@ -1,69 +0,0 @@ -#!/usr/bin/env python - -# -# Copyright 2019 Banco Bilbao Vizcaya Argentaria, S.A. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -import sys - -import click -import requests - -@click.group() -def kroute(): - pass - -@kroute.command() -@click.option("-c", "--command", nargs=1) -@click.option("-e", "--entrypoint", default="/bin/sh -c") -@click.option("-X", "--method", default="GET") -@click.option("--url", envvar='KAPOW_URL') -@click.argument("url_pattern", nargs=1) -@click.argument("command_file", required=False) -def add(url_pattern, entrypoint, command, method, url, command_file): - if command: - # Command is given inline - source = command - elif command_file is None: - # No command - source = "" - elif command_file == '-': - # Read commands from stdin - source = sys.stdin.read() - else: - # Read commands from a file - with open(command_file, 'r', encoding='utf-8') as handler: - source = handler.read() - - response = requests.post(f"{url}/routes", - json={"method": method, - "url_pattern": url_pattern, - "entrypoint": entrypoint, - "command": source}) - response.raise_for_status() - print(response.json()) - - -@kroute.command() -@click.option("--url", envvar='KAPOW_URL') -@click.argument("route-id") -def remove(route_id, url): - response = requests.delete(f"{url}/routes/{route_id}") - response.raise_for_status() - print(response.json()) - - -if __name__ == '__main__': - kroute() diff --git a/poc/bin/request b/poc/bin/request index 3268445..0ebf854 100755 --- a/poc/bin/request +++ b/poc/bin/request @@ -16,4 +16,4 @@ # limitations under the License. # -curl -sf ${KAPOW_URL}/connections/${KAPOW_CONNECTION}/request$1 +curl -sf "${KAPOW_URL}/handlers/${KAPOW_HANDLER_ID}/request$1" diff --git a/poc/bin/response b/poc/bin/response index c0961e4..75dc091 100755 --- a/poc/bin/response +++ b/poc/bin/response @@ -24,17 +24,17 @@ import requests @click.command() @click.option("--url", envvar='KAPOW_URL') -@click.option("--connection", envvar='KAPOW_CONNECTION') +@click.option("--handler-id", envvar='KAPOW_HANDLER_ID') @click.argument("path", nargs=1) @click.argument("value", required=False) -def response(url, connection, path, value): +def response(url, handler_id, path, value): if value is None: data = sys.stdin.buffer else: data = value.encode('utf-8') try: - response = requests.put(f"{url}/connections/{connection}/response{path}", + response = requests.put(f"{url}/handlers/{handler_id}/response{path}", data=data) except requests.exceptions.ConnectionError: return False diff --git a/poc/bin/static b/poc/bin/static index 1149bbb..5698741 100755 --- a/poc/bin/static +++ b/poc/bin/static @@ -15,7 +15,7 @@ else ;; *) response /status 200 - response /header/Content-Type "$(python -m mimetypes "$BASE/$REAL" | awk '/type:/ {print $2; exit 0}; !/type:/ {print "application/octet-stream"}')" + response /headers/Content-Type "$(python -m mimetypes "$BASE/$REAL" | awk '/type:/ {print $2; exit 0}; !/type:/ {print "application/octet-stream"}')" response /body < "$BASE/$REAL" esac fi diff --git a/poc/buildme b/poc/buildme new file mode 100755 index 0000000..c01ee09 --- /dev/null +++ b/poc/buildme @@ -0,0 +1,3 @@ +#!/bin/sh + +docker build -t bbva/kapow:0.1 . diff --git a/poc/examples/eval.pow b/poc/examples/eval.pow old mode 100644 new mode 100755 index 15fddb4..209be5d --- a/poc/examples/eval.pow +++ b/poc/examples/eval.pow @@ -16,4 +16,4 @@ # limitations under the License. # -kroute add -X POST '/eval' -c '$($(request /body) | response /stream)' +kapow route add -X POST '/eval' -c '$($(request /body) | response /stream)' diff --git a/poc/examples/eval.testme b/poc/examples/eval.testme new file mode 100755 index 0000000..a17bfc9 --- /dev/null +++ b/poc/examples/eval.testme @@ -0,0 +1,8 @@ +#!/usr/bin/env sh + +curl -X POST --data-binary @- http://localhost:8080/eval <(response /body) \ =(request /body) EOF -kroute add -X GET '/formats/input' -c 'pandoc --list-input-formats | response /body' -kroute add -X GET '/formats/output' -c 'pandoc --list-output-formats | grep -v pdf | response /body' +kapow route add -X GET '/formats/input' -c 'pandoc --list-input-formats | response /body' +kapow route add -X GET '/formats/output' -c 'pandoc --list-output-formats | grep -v pdf | response /body' diff --git a/poc/examples/pandoc/testme b/poc/examples/pandoc/testme new file mode 100755 index 0000000..e2fabd3 --- /dev/null +++ b/poc/examples/pandoc/testme @@ -0,0 +1,8 @@ +#!/usr/bin/env sh + +curl -X POST --data-binary @- http://localhost:8080/convert/markdown/man < @@ -27,8 +27,8 @@ kroute add / - <<-'EOF' HTML EOF -kroute add /save/magnet -e '/bin/bash -c' - <<-'EOF' - link=$(request /param/link) +kapow route add /save/magnet -e '/bin/bash -c' - <<-'EOF' + link=$(request /params/link) [ -z $link ] && response /status 400 && exit 0 watch_folder=/tmp @@ -37,7 +37,7 @@ kroute add /save/magnet -e '/bin/bash -c' - <<-'EOF' echo "d10:magnet-uri${#link}:${link}e" > "meta-${BASH_REMATCH[1]}.torrent" response /status 302 - response /header/Location /torrent/list + response /headers/Location /torrent/list EOF -kroute add /torrent/list -c 'response /body "Not Implemented Yet"' +kapow route add /torrent/list -c 'response /body "Not Implemented Yet"' diff --git a/resources/kapow.gif b/resources/kapow.gif new file mode 100644 index 0000000..953446d Binary files /dev/null and b/resources/kapow.gif differ diff --git a/spec/README.md b/spec/README.md index f9d18d4..7604e6d 100644 --- a/spec/README.md +++ b/spec/README.md @@ -5,45 +5,33 @@ Because we think that: -- UNIX is great and we love it -- The UNIX shell is great +- UNIX® is great and we love it +- The UNIX® shell is great - HTTP interfaces are convenient and everywhere - CGI is not a good way to mix them ## How? -So, how we can mix the **web** and the **shell**? Let's see... +So, how we can mix the **web** and the **shell**? Let's see... The **web** and the **shell** are two different beasts, both packed with history. There are some concepts in HTTP and the shell that **resemble each other**. -``` - +------------------------+-------------------------+ - | HTTP | SHELL | - +--------------+------------------------+-------------------------+ - | Input | POST form-encoding | Command line parameters | - | Parameters | GET parameters | Environment variables | - | | Headers | | - | | Serialized body (JSON) | | - +--------------+------------------------+-------------------------+ - | Data Streams | Response/Request Body | Stdin/Stdout/Stderr | - | | Websocket | Input/Output files | - | | Uploaded files | | - +--------------+------------------------+-------------------------+ - | Control | Status codes | Signals | - | | HTTP Methods | Exit Codes | - +--------------+------------------------+-------------------------+ -``` + | | HTTP | Shell | + |------------------------|--------------------------------------------------------------------------------|----------------------------------------------------| + | Input
Parameters | POST form-encoding
Get parameters
Headers
Serialized body (JSON) | Command line parameters
Environment variables | + | Data Streams | Response/Request Body
Websocket
Uploaded files | stdin/stdout/stderr
Input/Output files | + | Control | Status codes
HTTP Methods | Signals
Exit Codes | Any tool designed to give an HTTP interface to an existing shell command -**must map concepts of boths**. For example: +**must map concepts from both domains**. For example: - "GET parameters" to "Command line parameters" - "Headers" to "Environment variables" -- "Stdout" to "Response body" +- "stdout" to "Response body" Kapow! is not opinionated about the different ways you can map both worlds. Instead, it provides a concise set of tools, with a set of sensible defaults, @@ -52,8 +40,8 @@ allowing the user to express the desired mapping in an explicit way. ### Why not tool "X"? -All the alternatives we found are **rigid** about how they match between HTTP -and shell concepts. +All the alternatives we found are **rigid** about the way they match between +HTTP and shell concepts. * [shell2http](https://github.com/msoap/shell2http): HTTP-server to execute shell commands. Designed for development, prototyping or remote control. @@ -78,17 +66,17 @@ incapable in others. * Boilerplate * Custom code = More bugs -* Security issues (Command injection, etc) +* Security issues (command injection, etc) * Dependency on developers -* **"A programming language is low level when its programs require attention to - the irrelevant"** *Alan Perlis* -* **There is more Unix-nature in one line of shell script than there is in ten - thousand lines of C** *Master Foo* +* *"A programming language is low level when its programs require attention to + the irrelevant."
—Alan Perlis* +* *"There is more Unix-nature in one line of shell script than there is in ten + thousand lines of C."
—Master Foo* ### Why not CGI? -* CGI is also **rigid** about how it matches between HTTP and UNIX process +* CGI is also **rigid** about how it matches between HTTP and UNIX® process concepts. Notably CGI *meta-variables* are injected into the script's environment; this behavior can and has been exploited by nasty attacks such as [Shellshock](https://en.wikipedia.org/wiki/Shellshock_(software_bug)). @@ -101,17 +89,29 @@ incapable in others. ## What? We named it Kapow!. It is pronounceable, short and meaningless... like every -good UNIX command ;-) +good UNIX® command ;-) TODO: Definition TODO: Intro to Architecture +### Core Concepts + +In this section we are going to define several concepts that will be used frequently throughout the spec. + + +#### `entrypoint` + +The entrypoint definition matches *Docker*'s shell form of it. +Technically it's a string which is to be passed to the `command` (`/bin/bash -c` +by default) as the code to be interpreted or executed when attending requests. + + ### API Kapow! server interacts with the outside world only through its HTTP API. Any -program making the correct HTTP request to a Kapow! server, can change its +program making the correct HTTP request to a Kapow! server can change its behavior. Kapow! exposes two distinct APIs, a control API and a data API, described @@ -126,6 +126,7 @@ whole lifetime of the server. ## Design Principles +* We reuse conventions of well-established software projects, such as Docker. * All requests and responses will leverage JSON as the data encoding method. * The API calls responses will have two distinct parts: * The HTTP status code (e.g., `400`, which is a bad request). The target @@ -135,16 +136,16 @@ whole lifetime of the server. operating the client. The human can use this information to make a decision on how to proceed. * All successful API calls will return a representation of the *final* state - attained by the objects which have been addressed (requested, set or + attained by the objects which have been addressed (either requested, set or deleted). For instance, given this request: -``` +```http HTTP/1.1 GET /routes ``` an appropiate reponse may look like this: -``` +```http 200 OK Content-Type: application/json Content-Length: 189 @@ -169,7 +170,7 @@ Kapow! provides a way to control its internal state through these elements. ### Routes Routes are the mechanism that allows Kapow! to find the correct program to -respond to an external event (e.g. an incomming HTTP request). +respond to an external event (e.g. an incoming HTTP request). #### List routes @@ -180,8 +181,24 @@ Returns JSON data about the current routes. * **Method**: `GET` * **Success Responses**: * **Code**: `200 OK`
- **Content**: TODO -* **Sample Call**: TODO + **Content**:
+ ```json + [ + { + "method": "GET", + "url_pattern": "/hello", + "entrypoint": null, + "command": "echo Hello World | response /body" + }, + { + "method": "POST", + "url_pattern": "/bye", + "entrypoint": null, + "command": "echo Bye World | response /body" + } + ] + ``` +* **Sample Call**: `$ curl $KAPOW_URL/routes` * **Notes**: Currently all routes are returned; in the future, a filter may be accepted. @@ -193,7 +210,7 @@ Accepts JSON data that defines a new route to be appended to the current routes. * **Method**: `POST` * **Header**: `Content-Type: application/json` * **Data Params**:
- ``` + ```json { "method": "GET", "url_pattern": "/hello", @@ -202,10 +219,10 @@ Accepts JSON data that defines a new route to be appended to the current routes. } ``` * **Success Responses**: - * **Code**: `200 OK`
+ * **Code**: `201 Created`
**Header**: `Content-Type: application/json`
**Content**:
- ``` + ```json { "method": "GET", "url_pattern": "/hello", @@ -217,13 +234,35 @@ Accepts JSON data that defines a new route to be appended to the current routes. * **Error Responses**: * **Code**: `400 Malformed JSON` * **Code**: `400 Invalid Data Type` + * **Code**: `400 Invalid Route Spec` * **Code**: `400 Missing Mandatory Field`
**Header**: `Content-Type: application/json`
- **Content**: `{ "mandatory_fields": ["field1", "field2", "and so on"] }` -* **Sample Call**: TODO + **Content**:
+ ```json + { + "missing_mandatory_fields": [ + "url_pattern", + "command" + ] + } + ``` +* **Sample Call**:
+ ```sh + $ curl -X POST --data-binary @- $KAPOW_URL/routes < - ``` + ```json { "method": "GET", "url_pattern": "/hello", @@ -247,7 +286,7 @@ Accepts JSON data that defines a new route to be appended to the current routes. * **Code**: `200 OK`
**Header**: `Content-Type: application/json`
**Content**:
- ``` + ```json { "method": "GET", "url_pattern": "/hello", @@ -259,15 +298,38 @@ Accepts JSON data that defines a new route to be appended to the current routes. * **Error Responses**: * **Code**: `400 Malformed JSON` * **Code**: `400 Invalid Data Type` + * **Code**: `400 Invalid Route Spec` * **Code**: `400 Missing Mandatory Field`
**Header**: `Content-Type: application/json`
- **Content**: `{ "mandatory_fields": ["field1", "field2", "and so on"] }` + **Content**:
+ ```json + { + "missing_mandatory_fields": [ + "url_pattern", + "command" + ] + } + ``` * **Code**: `400 Invalid Index Type` -* **Sample Call**: TODO + * **Code**: `400 Index Already in Use` + * **Code**: `404 Invalid Index` + * **Code**: `404 Invalid Route Spec` +* **Sample Call**:
+ ```sh + $ curl -X PUT --data-binary @- $KAPOW_URL/routes < - **Content**: TODO + **Content**:
+ ```json + { + "method": "GET", + "url_pattern": "/hello", + "entrypoint": null, + "command": "echo Hello World | response /body", + "index": 0 + } + ``` * **Error Responses**: * **Code**: `404 Not Found` -* **Sample Call**: TODO +* **Sample Call**:
+ ```sh + $ curl -X DELETE $KAPOW_URL/routes/ROUTE_1f186c92_f906_4506_9788_a1f541b11d0f + ``` * **Notes**: @@ -331,6 +405,7 @@ following resource paths: │ ├─ request All information related to the HTTP request. Read-Only │ ├──── method Used HTTP Method (GET, POST) +│ ├──── host Host part of the URL │ ├──── path Complete URL path (URL-unquoted) │ ├──── matches Previously matched URL path parts │ │ └──── @@ -381,7 +456,7 @@ following resource paths: - Comment: That would provide read-only access to the value of the request header `Content-Type`. - Read a field from a form. - Scenario: A request generated by submitting this form:
- ``` + ```html
First name:

@@ -428,8 +503,11 @@ Returns the value of the requested resource path, or an error if the resource pa **Code**: `400 Invalid Resource Path`
**Notes**: Check the list of valid resource paths at the top of this section. * **Code**: `404 Not Found` -* **Sample Call**: TODO -* **Notes**: TODO +* **Sample Call**:
+ ```sh + $ curl /handlers/$KAPOW_HANDLER_ID/request/body + ``` +* **Notes**: The content may be empty. #### Overwrite the value of a resource @@ -447,7 +525,10 @@ Returns the value of the requested resource path, or an error if the resource pa * **Code**: `404 Handler Not Found` * **Code**: `404 Name Not Found`
**Notes**: Although the resource path is correct, no such name is present in the request. For instance, `/request/headers/Foo`, when no `Foo` header is not present in the request. -* **Sample Call**: +* **Sample Call**:
+ ```sh + $ curl -X --data-binary '

Hello!

' PUT /handlers/$KAPOW_HANDLER_ID/response/body + ``` * **Notes**: @@ -470,34 +551,169 @@ You can run it by ... Any compliant implementation of Kapow! must provide these commands: -### `kapow` +### `kapow server` -This implements the server, XXX +This is the master command, that shows the help if invoked without args, and +runs the sub-commands when provided to it. #### Example +```sh +$ kapow +Usage: kapow [OPTIONS] COMMAND [ARGS]... + +Options: + TBD + +Commands: + start starts a Kapow! server + route operates on routes +... +``` + + +### `kapow start` + +This command runs the Kapow! server, which is the core of Kapow!. If +run without parameters, it will run an unconfigured server. It can accept a path +to a `.pow` file, which is a shell script that contains commands to configure +the Kapow! server. + +The `.pow` can leverage the `kapow route` command, which is used to define a route. +The `kapow route` command needs a way to reach the Kapow! server, and for that, +`kapow` provides the `KAPOW_URL` variable in the environment of the +aforementioned shell script. + +Every time the kapow server receives a request, it will spawn a process to +handle it, according to the specified entrypoint, `/bin/sh -c` by default, and then +execute the specified command. This command is tasked with processing the +incoming request, and can leverage the `request` and `response` commands to +easily access the `HTTP Request` and `HTTP Response`, respectively. + +In order for `request` and `response` to do their job, they require a way to +reach the Kapow! server, as well as a way to identify the current request being +served. Thus, the Kapow! server adds the `KAPOW_URL` and `KAPOW_HANDLER_ID` to the +process' environment. + + +#### Example +```sh +$ kapow start /path/to/service.pow +``` ### `kapow route` +To serve an endpoint, you must first register it. + +`kapow route` registers/deregisters a route, which maps an +`HTTP` method and a URL pattern to the code that will handle the request. + +When registering, you can specify an *entrypoint*, which defaults to `/bin/sh -c`, +and an argument to it, the *command*. + +To deregister a route you must provide a *route_id*. + +**Notes**: + * The entrypoint definition matches *Docker*'s shell form of it. + * The index matches the way *netfilter*'s `iptables` handles rule numbering. + + +#### **Environment** +- `KAPOW_URL` + + +#### **Help** +```sh +$ kapow route --help +Usage: kapow route [OPTIONS] COMMAND [ARGS]... + +Options: + --help Show this message and exit. + +Commands: + add + remove +``` +```sh +$ kapow route add --help +Usage: kapow route add [OPTIONS] URL_PATTERN [COMMAND_FILE] + +Options: + -c, --command TEXT + -e, --entrypoint TEXT + -X, --method TEXT + --url TEXT + --help Show this message and exit. +``` +```sh +$ kapow route remove --help +Usage: kapow route remove [OPTIONS] ROUTE_ID + +Options: + --url TEXT + --help Show this message and exit. +``` #### Example - +```sh +kapow route add -X GET '/list/{ip}' -c 'nmap -sL $(request /matches/ip) | response /body' +``` ### `request` +Exposes the requests' resources. + + +#### **Environment** +- `KAPOW_URL` +- `KAPOW_HANDLER_ID` + #### Example +```sh +# Access the body of the request +request /body +``` ### `response` +Exposes the response's resources. + + +#### **Environment** +- `KAPOW_URL` +- `KAPOW_HANDLER_ID` + #### Example +```sh +# Write to the body of the response +echo 'Hello, World!' | response /body +``` ## An End-to-End Example +```sh +$ cat nmap.kpow +kapow route add -X GET '/list/{ip}' -c 'nmap -sL $(request /matches/ip) | response /body' +``` +```sh +$ kapow ./nmap.kapow +``` +```sh +$ curl $KAPOW_URL/list/127.0.0.1 +Starting Nmap 7.70 ( https://nmap.org ) at 2019-05-30 14:45 CEST +Nmap scan report for localhost (127.0.0.1) +Host is up (0.00011s latency). +Not shown: 999 closed ports +PORT STATE SERVICE +22/tcp open ssh + +Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds +``` ## Test Suite Notes