Skip to content

Core without XMPP types: protobuf boundary between the core and the frontends - #559

Draft
vitalyster wants to merge 12 commits into
masterfrom
core-callbacks-wip
Draft

vitalyster wants to merge 12 commits into
masterfrom
core-callbacks-wip

Conversation

@vitalyster

Copy link
Copy Markdown
Collaborator

Summary

First steps toward building the spectrum2 core (transport-protocol, libtransport, spectrum2, transport-plugin) without Swiften and Boost, so that spectrum2 can serve non-XMPP clients through a raw protobuf frontend. The XMPP and Slack frontends keep Swiften/Boost.

This PR covers three steps:

  1. Standard library instead of Boost. filesystem, regex, function, thread, foreach, lexical_cast, Boost.Locale lowercasing (now Util::toLowerUTF8) and base64.
  2. Core events are Event<Sig> callbacks instead of boost::signals2. Each event has a single listener; where a second listener existed, it chains to the first.
  3. Protobuf boundary between the core and the frontends. libtransport no longer uses XMPP stanza types:
    • users are strings (the bare JID for XMPP);
    • buddies and rooms are their legacy network names;
    • statuses, vCards, participants and messages are pbnetwork types.

It also adds next/, a standalone prototype of a protobuf frontend plus an IRC backend. It is off by default (ENABLE_NEXT).

Design

  • Frontend creates the User, RosterManager, UserManager and Conversation subclasses. It reports login results and carries raw XML in both directions (features.rawxml). The XMPP-specific send* and on* members are gone.
  • Messages are (WrapperMessage_Type, ConversationMessage) pairs: plain message, subject, attention, delivery receipt, and the three typing states. NetworkConversation is gone; the backend listens on Conversation::onMessageToSend.
  • Per-user output goes through hooks in the frontend's subclasses:
    • RosterManager: doSendBuddyStatus, doRequestAuthorization, isValidBuddyName
    • User: handleRoomJoined, sendCurrentPresence
    • UserManager: doRemoveUser
  • Core User: setReadyToConnect, onStatusChanged(pbnetwork::Status), and joinRoom / leaveRoom. Joins wait until the user is connected and are redone after a reconnect.
  • Core UserManager: loginUser (checks vip_only and enable_transport) and autoRegisterUser.
  • XMPP frontend. These moved out of the core:
    • presence aggregation, client capabilities and the connect timeout → XMPPUser
    • presence-driven login, subscriptions and the user-removal timer → XMPPUserManager
    • MUC, carbons, message caching and OOB media wrapping → the new XMPPConversation
    • PresenceOracle, raw-XML parsing, and JID↔legacy-name mapping (new jidutils)
  • Slack frontend no longer fakes XMPP presences and messages. It calls loginUser, joinRoom / leaveRoom and ConversationManager::handleMessageReceived directly.
  • AdminInterface::handleQuery(string) returns the reply. The admin JID check moved into XMPPFrontend.

Behaviour changes

  • Base64 decoding no longer strips trailing NUL bytes. Encoding is unchanged.
  • "Ready to connect" now fires once. Before, a client with already-known capabilities could cause two Login messages to the backend.
  • Raw-XML stanzas sent to the backend are copies. Before, the stored user presence got its "to" address rewritten.
  • Slack connects to the legacy network as soon as the user logs in, instead of after the 5-second capabilities timeout. It also no longer posts empty messages for typing notifications.

Not done yet

  • The Config parser to replace program_options.
  • Asio networking. The core still uses Swift for the event loop, timers and connections.
  • UserRegistry (XMPP server mode) still lives in libtransport.
  • The protobuf frontend inside spectrum2.
  • A CI build without Swiften.
  • libtransport, the frontends and the tests stay C++17: Boost 1.78 signals2 does not compile as C++20.

Testing

  • Steps 1–2: MSVC build OK, libtransport_test 155/155.
  • Step 3 and next/: not compiled locally; CI is the first check.
  • The unit tests are ported to the new API. Linux-only code (libpurple, the SQL backends, spectrum_manager) was reviewed by reading only.

@vitalyster
vitalyster marked this pull request as draft October 24, 2026 09:45
vitalyster and others added 5 commits October 24, 2026 14:44
…callbacks, next prototype

Context
-------
Goal: add a raw protobuf frontend to spectrum2 and build the core without
Swiften and without Boost, so spectrum2 can serve non-XMPP clients. The
XMPP and Slack frontends keep Swiften/Boost and become optional.

Agreed decisions (from planning):
- Core = transport-protocol, libtransport, spectrum2, transport-plugin and
  Swift-free backends.
- Wire format: reuse pbnetwork WrapperMessage (protocol.proto); spectrum2 is
  a backend multiplexer.
- Standalone Asio for the event loop; C++20; a hand-written INI+CLI parser
  instead of program_options; compile-time frontend registry instead of
  boost::dll.
- Frontend transport (TCP single/many client, WebSocket) is not decided yet.

State: steps 1 and 2 are done and verified. Step 3 (protobuf-only Frontend
interface) is started: the header is changed, its implementations are not.
A standalone "next" prototype (protobuf frontend + IRC backend) is added.
Later steps (Config parser, Asio networking, core value types replacing
Swift types, XMPP frontend as an adapter, ProtobufFrontend in spectrum2, CI
for a no-Swiften build) are not done.

THE TREE DOES NOT BUILD AS-IS: see "Step 3" below.

Step 1: standard library instead of Boost
-----------------------------------------
Replaced Boost utilities that have direct std equivalents or small
hand-written replacements:
- BOOST_FOREACH -> range-for; lexical_cast -> std::to_string / stoi (the
  admin interface keeps a helper, since it formats doubles without
  trailing zeros, like lexical_cast did).
- boost::filesystem -> std::filesystem; boost::regex -> std::regex;
  boost::function -> std::function; boost::thread/mutex -> std::thread/mutex.
- Util gains replaceAll, split, iStartsWith/iEndsWith, toLowerUTF8,
  generateUUID; base64 is hand-written.
- Base64 decode no longer strips trailing NUL bytes from decoded data. The
  encoding is unchanged, so passwords already stored decode the same.
- Buddy names are lowercased with Util::toLowerUTF8 (simple case mapping
  for Latin, Greek, Cyrillic, Armenian, Georgian; independent of process
  locale) instead of Boost.Locale. The libpurple backend uses the same
  function so conversation names still match the core's. Boost.Locale is
  removed from find_package and from NetworkPlugin's global locale setup.
- Config.h no longer pulls in foreach/format/assign/algorithm; frontends,
  backends and spectrum_manager include the headers they use.
- ThreadPool uses std::thread. Finished workers are joined, not detached.
  This also fixes an off-by-one in releaseThread (it accepted index
  MAX_THREADS).
- RosterManager drops boost::pool_allocator.
- C++ standard: transport-protocol, transport-plugin and Swift-free
  backends build as C++20 (headers stay C++17-clean). libtransport,
  frontends, the Swiften backend and tests stay C++17: Boost 1.78
  signals2 disconnect-by-slot does not compile as C++20. This goes away
  when Swiften leaves libtransport.

Step 2: core events as Event<Sig> lambdas, no signals2
------------------------------------------------------
Core events were boost::signals2 signals with many listeners and
disconnect-by-slot. Every core event has one real listener, so they are now
single-listener Event<Sig> members (include/transport/Event.h):
- Event<Sig> derives from std::function<Sig>. Calling an empty Event does
  nothing, so emit sites are a plain call: onX(args); with no if (onX)
  guard.
- Subscribe with a lambda; unsubscribe with nullptr. The old
  disconnect(boost::bind(...)) calls become "= nullptr".
- The constructors and assignment are written out, not inherited from
  std::function. Inheriting them let an Event be wrapped inside another
  std::function recursively; this crashed AdminInterfaceTest with a stack
  overflow before the fix.

Wiring changes:
- Component owns the presence path. It sets frontend->onPresenceReceived
  to feed PresenceOracle first (handleIncomingPresence), then forward to
  onGeneralPresenceReceived, which UserManager uses. This keeps the old
  order (oracle, then UserManager). PresenceOracle no longer takes a
  Frontend* and no longer connects to it. Component also forwards
  onAvailableChanged to the oracle.
- UserManager forwards frontend messages to AdminInterface (when set)
  after its own handling. AdminInterface no longer connects to the frontend
  itself.
- Where a second listener replaced an existing one, it now chains to the
  previous handler instead of overwriting it: tests that listen next to
  UserManager, the adhoc and Slack user managers (onUserCreated), and the
  XMPP and discoinfo listeners (onBackendConfigUpdated).
- Unchanged, still boost::bind: Swift::Timer onTick, Swift::Connection and
  ConnectionServer signals, DomainNameResolveQuery results. They are Swiften
  objects with signals2 disconnect-by-slot and are converted in the
  networking step.

Step 3 (in progress): protobuf-only Frontend interface
------------------------------------------------------
include/transport/Frontend.h no longer includes Swiften. The core talks
to frontends in pbnetwork types and plain string user ids:
- init() drops Swift::EventLoop and NetworkFactories.
- sendPresence -> sendBuddyStatus(user, buddy, pbnetwork::Status);
  sendVCard(user, pbnetwork::VCard); sendRosterRequest -> requestRoster;
  sendMessage(pbnetwork::ConversationMessage). sendIQ and
  sendCapabilitiesRequest are removed.
- createUser takes a std::string user instead of Swift::JID.
- Events use pbnetwork::VCard / Buddy / ConversationMessage.
  onPresenceReceived and onCapabilitiesReceived are replaced by
  onUserStatusChanged(pbnetwork::Status).

Not done: XMPPFrontend, SlackFrontend, Component, UserManager,
PresenceOracle, User, RosterManager and the tests still use the old
interface, so libtransport, spectrum2, frontends and tests do not compile.
The Step 2 wiring around onPresenceReceived must be redone on top of
onUserStatusChanged.

"next" prototype (ENABLE_NEXT, default OFF)
-------------------------------------------
A standalone subproject in next/, built only with -DENABLE_NEXT=ON. It
links only transport-protocol (protocol.pb.*) and JsonCpp, no Swiften,
no Boost, no libtransport. A first end-to-end test of the protobuf idea,
not the final design:
- common/wire: blocking TCP sockets, frames = 4-byte big-endian length +
  serialized WrapperMessage.
- common/client_json: one JSON object per line for client sockets,
  {"type": "TYPE_LOGIN", "payload": {...}} using protobuf JSON mapping.
  Payloads: Login, Logout, Connected, Disconnected, ConversationMessage.
- spectrum_next_frontend [client_port=5230] [backend_port=5231]: accepts
  clients and one backend, routes messages by userName. Any password is
  accepted.
- spectrum_next_irc [frontend_host] [frontend_port] [irc_host] [irc_port]:
  connects to the frontend as backend, one IRC connection per user,
  legacyName is the nick, buddyName is a nick or #channel (joined on the
  first message).
Thread per connection; no Asio yet.

Known issues and risks
----------------------
- Build is broken until Step 3 is finished (see above).
- next/common/wire.cpp uses SOCKET, INVALID_SOCKET and closesocket outside
  #ifdef _WIN32; it will not compile on Linux without typedefs.
- Chaining depends on setup order. Whoever installs first gets called
  first. The chained cases are onUserCreated and onBackendConfigUpdated.
  Replacing them with explicit subscriber lists, or a virtual hook, is the
  cleaner fix. Not done.
- Event silently ignores a call with no listener. A forgotten subscription
  is no longer an exception; tests are the only check.
- Linux-only code is not compiled here: libpurple, libcommuni, frotz,
  smstools3, spectrum_manager, MySQL/PQXX, and the log4cxx paths in
  protocol/Logging.cpp. Reviewed by reading only. The musl and clang CI
  images may reject the C++20 change; CI should confirm.

Verification
------------
- Steps 1 and 2 (before the Step 3 header change): MSVC (Visual Studio
  2026, Win32), RelWithDebInfo, unity build: all targets built with no
  errors; libtransport_test 155 of 155 OK (149 pre-existing + 6 new Util
  tests).
- Step 3 and next/: not built or tested.

History
-------
Squash of:
- de8b8b2b  Remove Swiften/Boost from the core, step by step: std library,
            Event callbacks (itself a squash of b0effdae, 1689df45, e58287a0)
- d97e3657  wip (Frontend.h protobuf interface, next/ prototype)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontend

The core (libtransport) no longer uses XMPP element types: no Swift::JID,
Presence, Message, IQ, VCard, RosterPayload, DiscoInfo or MUC payloads.
Users are strings (the bare JID for XMPP), buddies and rooms are legacy
names, statuses and messages are pbnetwork types. Swift is still used for
the event loop, timers and networking (next step: Asio), and by
UserRegistry (XMPP server mode, still in libtransport).

Swift:: uses in libtransport + include/transport: ~620 -> 75, all of them
networking, timers, the event loop and UserRegistry.

Core interface
--------------
- Frontend: creates User, RosterManager, UserManager and (new)
  Conversation; handleLoginResult (backend accepted/rejected a login);
  sendRawXML / onRawXMLReceived for features.rawxml; room list takes legacy
  room names. Events use pbnetwork::VCard / pbnetwork::Buddy.
  sendPresence, sendMessage, sendIQ, sendRosterRequest,
  sendCapabilitiesRequest, onPresenceReceived, onMessageReceived,
  onCapabilitiesReceived, onAvailableChanged are gone.
- User: getUserName(); setReadyToConnect(); onStatusChanged(pbnetwork::Status);
  joinRoom/leaveRoom (joins are postponed until connected and redone on
  reconnect) with handleRoomJoined hook; disconnectUser takes
  pbnetwork::ConnectionError. Presence aggregation, caps and the connecting
  timeout moved to XMPPUser.
- UserManager: loginUser (vip_only, enable_transport, create, onUserCreated)
  and autoRegisterUser; doRemoveUser hook. Presence-driven login,
  subscriptions, connect/disconnectUser and the remove timer moved to
  XMPPUserManager.
- Buddy/LocalBuddy: status is pbnetwork::StatusType; JIDs, presence stanzas
  and raw presences moved to XMPPRosterManager (generatePresenceStanzas,
  handleRawPresence) and jidutils (getBuddyJID, getSafeName,
  JIDToLegacyName, JIDToBuddy, buddyFlagsFromJID). RosterManager gets
  doSendBuddyStatus, doRequestAuthorization, isValidBuddyName hooks.
- Conversation: messages are (WrapperMessage_Type, ConversationMessage):
  TYPE_CONV_MESSAGE, TYPE_ROOM_SUBJECT_CHANGED, TYPE_ATTENTION,
  TYPE_CONV_MESSAGE_ACK, TYPE_BUDDY_TYPING/TYPED/STOPPED_TYPING. The core
  keeps the participant list (pbnetwork::Participant); handleMessage and
  doHandleParticipantChanged are implemented by the frontend's subclass.
  The backend listens on onMessageToSend (NetworkConversation is gone).
- ConversationManager: getConversationForMessage / handleMessageReceived
  (PM detection via ConversationMessage.pm + nickname).
- AdminInterface::handleQuery(string) returns the reply; the admin_jid
  check and the reply stanza moved to XMPPFrontend.
- NetworkPluginServer: no XMPP parser/serializer; chat states, receipts,
  attentions, subjects, xhtml/OOB media wrapping and timestamps are
  rendered by XMPPConversation.

XMPP frontend
-------------
- XMPPConversation: everything XMPP from the old Conversation (resources,
  MUC presences, subject ordering, caching, carbons, XEP-0356) plus the
  media wrapping from NetworkPluginServer.
- PresenceOracle moved here; XMPPFrontend feeds it and routes presences to
  XMPPUserManager as Component did before. Raw XML parsing/serializing
  (features.rawxml) moved here from NetworkPluginServer.

Slack frontend
--------------
- No more simulated XMPP presences/messages: SlackSession logs the user in
  via UserManager::loginUser, joins/leaves rooms via User::joinRoom /
  leaveRoom and sends Slack messages through
  ConversationManager::handleMessageReceived. SlackConversation posts
  TYPE_CONV_MESSAGE to Slack. Channel maps are keyed by legacy room name.

Behaviour changes
-----------------
- onReadyToConnect fires once: before, a client with known caps could
  trigger it twice (two Login messages to the backend).
- Raw-XML stanzas sent to the backend are copies; before, the user's
  presence held by the PresenceOracle got its "to" rewritten.
- Slack connects the legacy network at login instead of after the 5 s
  caps timeout, and posts only text messages (no empty chat-state posts).

Tests
-----
Unit tests ported: TestingFactory creates conversations through the
frontend; assertions on messages to the backend check pbnetwork types.

Not verified
------------
Not compiled or run here (no local builds); CI is the check. Linux-only
code reviewed by reading only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config.h no longer pulls them in through Boost headers; the Linux builds failed.
vitalyster and others added 7 commits October 24, 2026 15:15
…ine parser

transport-protocol no longer uses Boost. Config keeps its interface:

- ConfigValue (std::variant of string, int, bool, list, vector) replaces
  boost::program_options::variable_value; as<T>() throws on a type
  mismatch as before, so CONFIG_*_DEFAULTED and safeAs keep their meaning.
- inih (r62, BSD, vendored in protocol/inih) parses the config files. '#'
  starts a comment anywhere on a line, as with program_options; multi-line
  values are off and lines can be of any length.
- The registered variables are a table with the same keys, types and
  defaults. Storing follows program_options: the first value given is
  final (command line before the file, earlier loads before reload),
  defaults are replaced, a registered variable set twice in one file is an
  error, multi-value variables accumulate.
- CommandLine replaces program_options in Config::createFromArgs and in
  spectrum2's main(); same options, same "-x value", "--name=value" and
  positional config file forms, unknown options ignored.

Changes: syntax errors and invalid int/bool values make load() print the
error and return false instead of throwing out of it.

transport-protocol stops linking Boost; libtransport links it itself.
Files that got Boost headers through Config.h include them directly.
New tests cover comments, types, defaults, duplicates, $jid, reload and
CommandLine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ON by default. ENABLE_SLACK_FRONTEND is now a dependent option: it can
only be turned on together with ENABLE_XMPP_FRONTEND and is OFF otherwise.
With ENABLE_XMPP_FRONTEND OFF neither frontend is built, spectrum2
doesn't link them, and libtransport_test, which compiles the frontend
sources in, isn't built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IRC backend reads Login.legacyName as "nick@server[:port]", as the
libpurple IRC backend does; a bare nick still uses the server given on
the command line. Errors name the server it could not reach.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- client_json knows TYPE_JOIN_ROOM / TYPE_LEAVE_ROOM (pbnetwork::Room),
  so the frontend relays them to the backend.
- The IRC backend sends JOIN on JoinRoom (messages sent before the join is
  confirmed are queued, no second JOIN) and PART on LeaveRoom. A non-empty
  Login.password is sent as PASS before NICK (on Libera it identifies the
  nick to NickServ); PASS lines are not logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IRC backend tracks who is in each channel and sends
TYPE_PARTICIPANT_CHANGED (pbnetwork::Participant) for the NAMES list
(~ & @ % prefixes are PARTICIPANT_FLAG_MODERATOR, our own nick
PARTICIPANT_FLAG_ME), JOIN, PART, KICK (FLAG_KICKED), QUIT (gone from
every channel the nick was in) and NICK (newname); gone is STATUS_NONE
as elsewhere in Spectrum. The frontend relays them to the user and
client_json knows the Participant payload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/login, /api/send, /api/logout and a chunked text/event-stream
at GET /api/events that resumes from Last-Event-ID. Meant to sit behind
nginx for HTTPS; listens on 127.0.0.1 by default. A session with no open
event stream for 90 s is logged out at the backend.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant