docs: 📝 Updated documentation for Lysand 0.4

Updated outdated documentation in preparation for the release of Lysand 0.4
This commit is contained in:
Jesse Wierzbinski 2024-04-14 13:33:02 -10:00
parent 5e9e05826b
commit 2e2b91d7a5
No known key found for this signature in database
6 changed files with 153 additions and 222 deletions

View file

@ -12,14 +12,13 @@ Lysand is built using the following technologies:
- [Bun](https://bun.sh) - A JavaScript runtime similar to Node.js, but faster and with more features
- [PostgreSQL](https://www.postgresql.org/) - A relational database
- [`pg_uuidv7`](https://github.com/fboulnois/pg_uuidv7) - A PostgreSQL extension that provides a UUIDv7 data type
- [UnoCSS](https://unocss.dev) - A utility-first CSS framework, used for the login page
- [Nuxt](https://nuxt.com/) - A Vue.js framework, used for the frontend
- [Docker](https://www.docker.com/) - A containerization platform, used for development and deployment
- [Sharp](https://sharp.pixelplumbing.com/) - An image processing library, used for fast image resizing and converting
- [TypeScript](https://www.typescriptlang.org/) - A typed superset of JavaScript
- [ESLint](https://eslint.org/) - A JavaScript linter
- [Prettier](https://prettier.io/) - A code formatter
## Getting Started
To get started, please follow these steps:
1. Fork the repository, clone it on your local system and make your own branch
@ -39,50 +38,9 @@ git clone https://github.com/lysand-org/lysand.git
bun install
```
3. Set up a PostgreSQL database, using the `pg_uuidv7` extension
3. Set up a PostgreSQL database (you need a special extension, please look at [the database documentation](database.md))
You may use the following [Dockerfile](Postgres.Dockerfile) to set it up:
```Dockerfile
# Use the latest Postgres Docker image based on Alpine
FROM postgres:alpine
# Set working directory
WORKDIR /usr/src/app
# Install curl
RUN apk add --no-cache curl
RUN cd "$(mktemp -d)" \
&& curl -LO "https://github.com/fboulnois/pg_uuidv7/releases/download/v1.3.0/{pg_uuidv7.tar.gz,SHA256SUMS}" \
&& tar xf pg_uuidv7.tar.gz \
&& sha256sum -c SHA256SUMS \
&& PG_MAJOR=$(pg_config --version | sed 's/^.* \([0-9]\{1,\}\).*$/\1/') \
&& cp "$PG_MAJOR/pg_uuidv7.so" "$(pg_config --pkglibdir)" \
&& cp sql/pg_uuidv7--1.3.sql pg_uuidv7.control "$(pg_config --sharedir)/extension"
# Add a script to run the CREATE EXTENSION command
RUN echo '#!/bin/sh\npsql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "CREATE EXTENSION pg_uuidv7;"' > /docker-entrypoint-initdb.d/init.sh
# Make the entrypoint script executable
RUN chmod +x /docker-entrypoint-initdb.d/init.sh
```
4. Copy the `config.toml.example` file to `config.toml` and fill in the values (you can leave most things to the default, but you will need to configure things such as the database connection)
> [!WARNING]
> You should disable Prisma Redis caching while developing, as it can mess up tests
5. Generate the Prisma client:
```bash
bun prisma generate
```
6. Run migrations:
```bash
bun migrate
```
4. Copy the `config/config.toml.example` file to `config/config.toml` and edit it to set up the database connection and other settings.
## Testing your changes
@ -93,14 +51,14 @@ bun dev
If your port number is lower than 1024, you may need to run the command as root.
### Running the Vite server
### Running the FE
To start the Vite server, run:
To start the frontend server, run:
```sh
bun vite:dev
bun fe:dev
```
This should be run in a separate terminal window. The Vite server is used to serve the frontend assets and to provide hot module reloading.
This should be run in a separate process as the server.
## Running tests
@ -109,37 +67,32 @@ To run the tests, run:
bun test
```
The tests are located in the `tests/` directory and follow a Jest-like syntax. The server does not need to be started before running the tests, as the tests will spawn their own Lysand server instance.
The tests are located in the `tests/` directory and follow a Jest-like syntax. The server should be shut down before running the tests.
## Code style
We use ESLint and Prettier to enforce a consistent code style. To check if your code is compliant, run:
We use [Biome](https://biomejs.dev) to enforce a consistent code style. To check if your code is compliant, run:
```sh
bun lint
bunx @biomejs/biome check .
```
To automatically fix the issues, run:
```sh
bun lint --fix
bunx @biomejs/biome check . --apply
```
You should have the [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) and [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) extensions installed in VSCode, if you use it. From the ESLint extension, you can automatically fix the issues with `Ctrl+Shift+P` and `ESLint: Fix all auto-fixable Problems`.
ESLint and Prettier are also integrated in the CI pipeline, so your code will be automatically checked when you push it. If the pipeline fails, you will need to fix the issues before your pull request can be merged.
Code style such as brackets, spaces/tabs, etc are enforced by Prettier's ESLint plugin. You can find the simple configuration in the `.prettierrc` file.
You can also install the Biome Visual Studio Code extension and have it format your code automatically on save.
### ESLint rules
ESLint errors should not be ignored, except if they are false positives, in which case you can use a comment to disable the rule for the line or the file. If you need to disable a rule, please add a comment explaining why.
Linting should not be ignored, except if they are false positives, in which case you can use a comment to disable the rule for the line or the file. If you need to disable a rule, please add a comment explaining why.
TypeScript errors should be ignored with `// @ts-expect-error` comments, as well as with a reason for being ignored.
### Commit messages
We use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) for our commit messages. This allows us to automatically generate the changelog and the version number.
> **Note**: I don't actually enforce this rule, but it would be great if you could follow it.
We use [Conventional Commits](https://www.conventionalcommits.org) for our commit messages. This allows us to automatically generate the changelog and the version number.
### Pull requests
@ -151,13 +104,15 @@ We use Bun's integrated testing system to write tests. You can find more informa
Tests **should** be written for all API routes and all functions that are not trivial. If you are not sure whether you should write a test for something, you probably should.
To help with the creation of tests, you may find [GitHub Copilot](https://copilot.github.com/) useful (or some of its free alternatives like [Codeium](https://codeium.com/)). Please do not blindly copy the code that it generates, but use it as a starting point for your own tests. I recognize that writing tests is very tedious, which is why LLMs can come in handy.
#### Adding per-route tests
To add tests for a route, create a `route_file_name.test.ts` file in the same directory as the route itself. See [this example](/server/api/api/v1/timelines/home.test.ts) for help writing tests.
### Writing documentation
Documentation for the Lysand protocol is available on [lysand.org](https://lysand.org/). If you are thinking of modifying the protocol, please make sure to send a pull request over there to get it approved and merged before you send your pull request here.
This project should not need much documentation, but if you think that something needs to be documented, please add it to the README or contribution guide.
This project should not need much documentation, but if you think that something needs to be documented, please add it to the README, docs or contribution guide.
## Reporting bugs

158
README.md
View file

@ -2,11 +2,11 @@
<a href="https://lysand.org"><img src="https://cdn-web.cpluspatch.com/lysand.webp" alt="Lysand Logo" height=130></a>
</p>
![Postgres](https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white) ![Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=for-the-badge&logo=bun&logoColor=white) ![VS Code Insiders](https://img.shields.io/badge/VS%20Code%20Insiders-35b393.svg?style=for-the-badge&logo=visual-studio-code&logoColor=white) ![TypeScript](https://img.shields.io/badge/typescript-%23007ACC.svg?style=for-the-badge&logo=typescript&logoColor=white) ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) ![ESLint](https://img.shields.io/badge/ESLint-4B3263?style=for-the-badge&logo=eslint&logoColor=white) [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa?style=for-the-badge)](code_of_conduct.md)
![Postgres](https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white) ![Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=for-the-badge&logo=bun&logoColor=white) ![VS Code Insiders](https://img.shields.io/badge/VS%20Code%20Insiders-35b393.svg?style=for-the-badge&logo=visual-studio-code&logoColor=white) ![TypeScript](https://img.shields.io/badge/typescript-%23007ACC.svg?style=for-the-badge&logo=typescript&logoColor=white) ![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) ![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge&logo=docker&logoColor=white) [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa?style=for-the-badge)](code_of_conduct.md)
## What is this?
This is a project to create a federated social network based on the [Lysand](https://lysand.org) protocol. It is currently in alpha phase, with basic federation and almost complete Mastodon API support.
This is a project to create a federated social network based on the [Lysand](https://lysand.org) protocol. It is currently in beta phase, with basic federation and almost complete Mastodon API support.
This project aims to be a fully featured social network, with a focus on privacy, security, and performance. It implements the Mastodon API for support with clients that already support Mastodon or Pleroma.
@ -15,7 +15,7 @@ This project aims to be a fully featured social network, with a focus on privacy
## Features
- [x] Inbound federation
- [x] Federation (partial)
- [x] Hyper fast (thousands of HTTP requests per second)
- [x] S3 or local media storage
- [x] Deduplication of uploaded files
@ -26,12 +26,11 @@ This project aims to be a fully featured social network, with a focus on privacy
- [x] Automatic image conversion to WebP or other formats
- [x] Scripting-compatible CLI with JSON and CSV outputs
- [ ] Moderation tools
- [ ] Full Mastodon API support
- [ ] Outbound federation
- [x] Mastodon API support (partial)
## Screenshots
Here are some screenshots of Lysand's built-in web client. This client is built with Vue, and serves to allow users to log in or register. It is kept to only the actions that cannot be done with a Mastodon client, to simplify development. In the future, there may be a full-featured client.
Here are some screenshots of Lysand's built-in web client. This client is built with Nuxt, and serves to allow users to log in or register. It is kept to only the actions that cannot be done with a Mastodon client, to simplify development. In the future, there will be a full-featured client.
### On Desktop
@ -114,130 +113,7 @@ Lysand is extremely fast and can handle thousands of HTTP requests per second on
## How do I run it?
### Requirements
- The [Bun Runtime](https://bun.sh), version 1.0.30 or later (usage of the latest version is recommended)
- A PostgreSQL database
- (Optional but recommended) A Linux-based operating system
- (Optional if you want search) A working Meiliseach instance
> [!WARNING]
> Lysand has not been tested on Windows or MacOS. It is recommended to use a Linux-based operating system to run Lysand.
>
> We will not be offerring support to Windows or MacOS users. If you are using one of these operating systems, please use a virtual machine or container to run Lysand.
### Installation
1. Clone this repository
```bash
git clone https://github.com/lysand-org/lysand.git
```
2. Install the dependencies
```bash
bun install
```
3. Set up a PostgreSQL database, using the `pg_uuidv7` extension
Lysand offers a pre-made Docker image for PostgreSQL with the extension already installed. Use `ghcr.io/lysand-org/postgres:main` as your Docker image name to use it.
You may otherwise use this [Dockerfile](Postgres.Dockerfile) to set it up.
1. Copy the `config.toml.example` file to `config.toml` and fill in the values (you can leave most things to the default, but you will need to configure things such as the database connection)
2. Run migrations:
```bash
bun migrate
```
6. (If you want search)
Create a Meilisearch instance (using Docker is recommended). For a [`docker-compose`] file, copy the `meilisearch` service from the [`docker-compose.yml`](docker-compose.yml) file.
Set up Meiliseach's API key by passing the `MEILI_MASTER_KEY` environment variable to the server. Then, enale and configure search in the config file.
7. Build everything:
```bash
bun prod-build
```
You may now start the server with `bun start`. It lives in the `dist/` directory, all the other code can be removed from this point onwards.
In fact, the `bun start` script merely runs `bun run dist/index.js --prod`!
### Running
To run the server, simply run the following command:
```bash
bun start
```
### Using the CLI
Lysand includes a built-in CLI for managing the server. To use it, simply run the following command:
```bash
bun cli help
```
If you are running a production build, you will need to run `bun run dist/cli.js` or `./entrypoint.sh cli` instead.
You can use the `help` command to see a list of available commands. These include creating users, deleting users and more. Each command also has a `--help,-h` flag that you can use to see more information about the command.
#### Scripting with the CLI
Some CLI commands that return data as tables can be used in scripts. To convert them to JSON or CSV, some commands allow you to specify a `--format` flag that can be either `"json"` or `"csv"`. See `bun cli help` or `bun cli <command> -h` for more information.
Flags can be used in any order and anywhere in the script (except for the `bun cli` command itself). The command arguments themselves must be in the correct order, however.
### Rebuilding the Search Index
You may use the `bun cli index rebuild` command to automatically push all posts and users to Meilisearch, if it is configured. This is useful if you have just set up Meilisearch, or if you accidentally deleted something.
### Using Database Commands
The `bun prisma` commands allows you to use Prisma commands without needing to add in environment variables for the database config. Just run Prisma commands as you would normally, replacing `bunx prisma` with `bun prisma`.
## With Docker
> [!NOTE]
> Docker is currently broken, as Bun with Prisma does not work well with Docker yet for unknown reasons. The following instructions are for when this is fixed.
>
> These instructions will probably also work with Podman and other container runtimes.
You can also run Lysand using Docker. To do so, you can:
1. Acquire the Postgres Dockerfile from above
2. Use this repository's [`docker-compose.yml`](docker-compose.yml) file
3. Create the `lysand-net` docker network:
```bash
docker network create lysand-net
```
1. Fill in the config file (see [Installation](#installation))
2. Run the following command:
```bash
docker-compose up -d
```
You may need root privileges to run Docker commands.
### Running CLI commands inside Docker
You can run CLI commands inside Docker using the following command:
```bash
sudo docker exec -it lysand sh entrypoint.sh cli ...
```
### Running migrations inside Docker
You can run migrations inside Docker using the following command (if needed):
```bash
sudo docker exec -it lysand sh entrypoint.sh prisma migrate deploy
```
Please see the [installation guide](docs/installation.md) for more information on how to install Lysand.
## Contributing
@ -250,9 +126,6 @@ Contributions are welcome! Please see the [CONTRIBUTING.md](CONTRIBUTING.md) fil
## Federation
> [!WARNING]
> Federation has not been tested outside of automated tests. It is not recommended to use this software in production.
The following extensions are currently supported or being worked on:
- `org.lysand:custom_emojis`: Custom emojis
@ -304,23 +177,24 @@ Working endpoints are:
- `/api/v1/mutes`
- `/api/v2/media`
- `/api/v1/notifications`
Tests needed but completed:
- `/api/v1/media/:id`
- `/api/v2/media`
- `/api/v1/favourites`
- `/api/v1/accounts/:id/followers`
- `/api/v1/accounts/:id/following`
- `/api/v2/search`
- `/api/v1/follow_requests`
- `/api/v1/follow_requests/:account_id/authorize`
- `/api/v1/follow_requests/:account_id/reject`
- `/api/v1/statuses/:id/mute`
- `/api/v1/statuses/:id/unmute`
- `/api/v1/statuses/:id` (`PUT`)
- `/api/v1/statuses/:id/source`
Endpoints left:
- `/api/v1/reports`
- `/api/v1/accounts/:id/lists`
- `/api/v1/follow_requests`
- `/api/v1/follow_requests/:account_id/authorize`
- `/api/v1/follow_requests/:account_id/reject`
- `/api/v1/follow_suggestions`
- `/api/v1/domain_blocks` (`GET`, `POST`, `DELETE`)
- `/api/v2/filters` (`GET`, `POST`)
@ -343,11 +217,7 @@ Endpoints left:
- `/api/v1/statuses/:id/translate`
- `/api/v1/statuses/:id/bookmark`
- `/api/v1/statuses/:id/unbookmark`
- `/api/v1/statuses/:id/mute`
- `/api/v1/statuses/:id/unmute`
- `/api/v1/statuses/:id` (`PUT`)
- `/api/v1/statuses/:id/history`
- `/api/v1/statuses/:id/source`
- `/api/v1/polls/:id`
- `/api/v1/polls/:id/votes`
- `/api/v1/scheduled_statuses`
@ -379,7 +249,7 @@ Endpoints left:
- `/api/v1/announcements/:id/reactions/:name` (`PUT`, `DELETE`)
- Admin API
WebSocket Streaming API also needed to be added (and push notifications)
WebSocket Streaming API also needs to be added (and push notifications)
## License

24
docs/cli.md Normal file
View file

@ -0,0 +1,24 @@
# Lysand CLI
Lysand includes a built-in, scripting-compatible CLI that can be used to manage the server. This CLI can be used to create and delete users, manage the database and more. It can also output data in JSON or CSV format, making it easy to use in scripts.
## Using the CLI
Lysand includes a built-in CLI for managing the server. To use it, simply run the following command:
```bash
# Development
bun cli help
# Source installs
bun run dist/cli.js help
# Docker
docker compose exec -it lysand /bin/sh /app/entrypoint.sh cli help
```
You can use the `help` command to see a list of available commands. These include creating users, deleting users and more. Each command also has a `--help,-h` flag that you can use to see more information about the command.
## Scripting with the CLI
Some CLI commands that return data as tables can be used in scripts. To convert them to JSON or CSV, some commands allow you to specify a `--format` flag that can be either `"json"` or `"csv"`. See `bun cli help` or `bun cli <command> -h` for more information.
Flags can be used in any order and anywhere in the script (except for the `bun cli` command itself). The command arguments themselves must be in the correct order, however.Z

10
docs/database.md Normal file
View file

@ -0,0 +1,10 @@
# Installing the database
Lysand uses a special PostgreSQL extension called `pg_uuidv7` to generate UUIDs. This extension is required for Lysand to work properly. To install it, you can either use the pre-made Docker image or install it manually.
## Using the Docker image
Lysand offers a pre-made Docker image for PostgreSQL with the extension already installed. Use `ghcr.io/lysand-org/postgres:main` as your Docker image name to use it.
## Manual installation

84
docs/installation.md Normal file
View file

@ -0,0 +1,84 @@
# Installation
### Requirements
- The [Bun Runtime](https://bun.sh), version 1.0.30 or later (usage of the latest version is recommended)
- A PostgreSQL database
- (Optional but recommended) A Linux-based operating system
- (Optional if you want search) A working Meiliseach instance
> [!WARNING]
> Lysand has not been tested on Windows or MacOS. It is recommended to use a Linux-based operating system to run Lysand.
>
> We will not be offerring support to Windows or MacOS users. If you are using one of these operating systems, please use a virtual machine or container to run Lysand.
### With Docker
Docker is the recommended way to run Lysand (podman should also work). To run Lysand with Docker, follow these steps:
1. Download the `docker-compose.yml` file from the repository
```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/lysand-org/lysand/main/docker-compose.yml
```
2. Edit the `docker-compose.yml` file to set up the database connection and other settings
3. Download the `config.toml.example` file from the repository
```bash
curl -o config.toml.example https://raw.githubusercontent.com/lysand-org/lysand/main/config.toml.example
```
4. Edit the `config.toml.example` file to set up the database connection and other settings, then place it inside `config/` (create the `config/` directory if it does not exist)
5. Run the following command to start the server:
```bash
docker-compose up
```
You may need root privileges to run Docker commands.
### From Source
1. Clone this repository
```bash
git clone https://github.com/lysand-org/lysand.git
```
2. Install the dependencies
```bash
bun install
```
1. Set up a PostgreSQL database (you need a special extension, please look at [the database documentation](database.md))
2. (If you want search)
Create a Meilisearch instance (using Docker is recommended). For a [`docker-compose`] file, copy the `meilisearch` service from the [`docker-compose.yml`](docker-compose.yml) file.
Set up Meiliseach's API key by passing the `MEILI_MASTER_KEY` environment variable to the server. Then, enable and configure search in the config file.
3. Build everything:
```bash
bun prod-build
```
4. Copy the `config.toml.example` file to `config.toml` inside `dist/config/` and fill in the values (you can leave most things to the default, but you will need to configure things such as the database connection)
You may now start the server with `bun start`. It lives in the `dist/` directory, all the other code can be removed from this point onwards.
### Running the Server
Database migrations are run automatically on startup.
You may use the environment variables `NO_COLORS=true` and `NO_FANCY_DATES=true` to disable colors and date formatting in the console logs: the file logs will never have colors or fancy dates.
Please see the [CLI documentation](cli.md) for more information on how to use the CLI.
### Updating the server
Updating the server is as simple as pulling the latest changes from the repository and running `bun prod-build` again. You may need to run `bun install` again if there are new dependencies.
For Docker, you can run `docker-compose pull` to update the Docker images.
Sometimes, new configuration options are added to `config.example.toml`. If you see a new option in the example file, you should add it to your `config.toml` file.

View file

@ -4,7 +4,7 @@ import type { MatchedRoute } from "bun";
import { type Config, config } from "config-manager";
import { LogLevel, type LogManager, type MultiLogManager } from "log-manager";
import { RequestParser } from "request-parser";
import { type ZodType, z } from "zod";
import type { ZodType, z } from "zod";
import { fromZodError } from "zod-validation-error";
import {
type AuthData,
@ -59,18 +59,6 @@ export interface APIRouteExports {
default: RouteHandler<APIRouteMetadata, z.AnyZodObject>;
}
const exampleZodSchema = z.object({
allowedMethods: z.array(z.string()),
ratelimits: z.object({
max: z.number(),
duration: z.number(),
}),
route: z.string(),
auth: z.object({
required: z.boolean(),
}),
});
export const processRoute = async (
matchedRoute: MatchedRoute,
request: Request,