Repository navigation
Core without XMPP types: protobuf boundary between the core and the frontends - #559
Draft
vitalyster wants to merge 12 commits into
Draft
vitalyster wants to merge 12 commits into
vitalyster wants to merge 12 commits into
Conversation
vitalyster
marked this pull request as draft
October 24, 2026 09:45
…callbacks, next prototype
Context
-------
Goal: add a raw protobuf frontend to spectrum2 and build the core without
Swiften and without Boost, so spectrum2 can serve non-XMPP clients. The
XMPP and Slack frontends keep Swiften/Boost and become optional.
Agreed decisions (from planning):
- Core = transport-protocol, libtransport, spectrum2, transport-plugin and
Swift-free backends.
- Wire format: reuse pbnetwork WrapperMessage (protocol.proto); spectrum2 is
a backend multiplexer.
- Standalone Asio for the event loop; C++20; a hand-written INI+CLI parser
instead of program_options; compile-time frontend registry instead of
boost::dll.
- Frontend transport (TCP single/many client, WebSocket) is not decided yet.
State: steps 1 and 2 are done and verified. Step 3 (protobuf-only Frontend
interface) is started: the header is changed, its implementations are not.
A standalone "next" prototype (protobuf frontend + IRC backend) is added.
Later steps (Config parser, Asio networking, core value types replacing
Swift types, XMPP frontend as an adapter, ProtobufFrontend in spectrum2, CI
for a no-Swiften build) are not done.
THE TREE DOES NOT BUILD AS-IS: see "Step 3" below.
Step 1: standard library instead of Boost
-----------------------------------------
Replaced Boost utilities that have direct std equivalents or small
hand-written replacements:
- BOOST_FOREACH -> range-for; lexical_cast -> std::to_string / stoi (the
admin interface keeps a helper, since it formats doubles without
trailing zeros, like lexical_cast did).
- boost::filesystem -> std::filesystem; boost::regex -> std::regex;
boost::function -> std::function; boost::thread/mutex -> std::thread/mutex.
- Util gains replaceAll, split, iStartsWith/iEndsWith, toLowerUTF8,
generateUUID; base64 is hand-written.
- Base64 decode no longer strips trailing NUL bytes from decoded data. The
encoding is unchanged, so passwords already stored decode the same.
- Buddy names are lowercased with Util::toLowerUTF8 (simple case mapping
for Latin, Greek, Cyrillic, Armenian, Georgian; independent of process
locale) instead of Boost.Locale. The libpurple backend uses the same
function so conversation names still match the core's. Boost.Locale is
removed from find_package and from NetworkPlugin's global locale setup.
- Config.h no longer pulls in foreach/format/assign/algorithm; frontends,
backends and spectrum_manager include the headers they use.
- ThreadPool uses std::thread. Finished workers are joined, not detached.
This also fixes an off-by-one in releaseThread (it accepted index
MAX_THREADS).
- RosterManager drops boost::pool_allocator.
- C++ standard: transport-protocol, transport-plugin and Swift-free
backends build as C++20 (headers stay C++17-clean). libtransport,
frontends, the Swiften backend and tests stay C++17: Boost 1.78
signals2 disconnect-by-slot does not compile as C++20. This goes away
when Swiften leaves libtransport.
Step 2: core events as Event<Sig> lambdas, no signals2
------------------------------------------------------
Core events were boost::signals2 signals with many listeners and
disconnect-by-slot. Every core event has one real listener, so they are now
single-listener Event<Sig> members (include/transport/Event.h):
- Event<Sig> derives from std::function<Sig>. Calling an empty Event does
nothing, so emit sites are a plain call: onX(args); with no if (onX)
guard.
- Subscribe with a lambda; unsubscribe with nullptr. The old
disconnect(boost::bind(...)) calls become "= nullptr".
- The constructors and assignment are written out, not inherited from
std::function. Inheriting them let an Event be wrapped inside another
std::function recursively; this crashed AdminInterfaceTest with a stack
overflow before the fix.
Wiring changes:
- Component owns the presence path. It sets frontend->onPresenceReceived
to feed PresenceOracle first (handleIncomingPresence), then forward to
onGeneralPresenceReceived, which UserManager uses. This keeps the old
order (oracle, then UserManager). PresenceOracle no longer takes a
Frontend* and no longer connects to it. Component also forwards
onAvailableChanged to the oracle.
- UserManager forwards frontend messages to AdminInterface (when set)
after its own handling. AdminInterface no longer connects to the frontend
itself.
- Where a second listener replaced an existing one, it now chains to the
previous handler instead of overwriting it: tests that listen next to
UserManager, the adhoc and Slack user managers (onUserCreated), and the
XMPP and discoinfo listeners (onBackendConfigUpdated).
- Unchanged, still boost::bind: Swift::Timer onTick, Swift::Connection and
ConnectionServer signals, DomainNameResolveQuery results. They are Swiften
objects with signals2 disconnect-by-slot and are converted in the
networking step.
Step 3 (in progress): protobuf-only Frontend interface
------------------------------------------------------
include/transport/Frontend.h no longer includes Swiften. The core talks
to frontends in pbnetwork types and plain string user ids:
- init() drops Swift::EventLoop and NetworkFactories.
- sendPresence -> sendBuddyStatus(user, buddy, pbnetwork::Status);
sendVCard(user, pbnetwork::VCard); sendRosterRequest -> requestRoster;
sendMessage(pbnetwork::ConversationMessage). sendIQ and
sendCapabilitiesRequest are removed.
- createUser takes a std::string user instead of Swift::JID.
- Events use pbnetwork::VCard / Buddy / ConversationMessage.
onPresenceReceived and onCapabilitiesReceived are replaced by
onUserStatusChanged(pbnetwork::Status).
Not done: XMPPFrontend, SlackFrontend, Component, UserManager,
PresenceOracle, User, RosterManager and the tests still use the old
interface, so libtransport, spectrum2, frontends and tests do not compile.
The Step 2 wiring around onPresenceReceived must be redone on top of
onUserStatusChanged.
"next" prototype (ENABLE_NEXT, default OFF)
-------------------------------------------
A standalone subproject in next/, built only with -DENABLE_NEXT=ON. It
links only transport-protocol (protocol.pb.*) and JsonCpp, no Swiften,
no Boost, no libtransport. A first end-to-end test of the protobuf idea,
not the final design:
- common/wire: blocking TCP sockets, frames = 4-byte big-endian length +
serialized WrapperMessage.
- common/client_json: one JSON object per line for client sockets,
{"type": "TYPE_LOGIN", "payload": {...}} using protobuf JSON mapping.
Payloads: Login, Logout, Connected, Disconnected, ConversationMessage.
- spectrum_next_frontend [client_port=5230] [backend_port=5231]: accepts
clients and one backend, routes messages by userName. Any password is
accepted.
- spectrum_next_irc [frontend_host] [frontend_port] [irc_host] [irc_port]:
connects to the frontend as backend, one IRC connection per user,
legacyName is the nick, buddyName is a nick or #channel (joined on the
first message).
Thread per connection; no Asio yet.
Known issues and risks
----------------------
- Build is broken until Step 3 is finished (see above).
- next/common/wire.cpp uses SOCKET, INVALID_SOCKET and closesocket outside
#ifdef _WIN32; it will not compile on Linux without typedefs.
- Chaining depends on setup order. Whoever installs first gets called
first. The chained cases are onUserCreated and onBackendConfigUpdated.
Replacing them with explicit subscriber lists, or a virtual hook, is the
cleaner fix. Not done.
- Event silently ignores a call with no listener. A forgotten subscription
is no longer an exception; tests are the only check.
- Linux-only code is not compiled here: libpurple, libcommuni, frotz,
smstools3, spectrum_manager, MySQL/PQXX, and the log4cxx paths in
protocol/Logging.cpp. Reviewed by reading only. The musl and clang CI
images may reject the C++20 change; CI should confirm.
Verification
------------
- Steps 1 and 2 (before the Step 3 header change): MSVC (Visual Studio
2026, Win32), RelWithDebInfo, unity build: all targets built with no
errors; libtransport_test 155 of 155 OK (149 pre-existing + 6 new Util
tests).
- Step 3 and next/: not built or tested.
History
-------
Squash of:
- de8b8b2b Remove Swiften/Boost from the core, step by step: std library,
Event callbacks (itself a squash of b0effdae, 1689df45, e58287a0)
- d97e3657 wip (Frontend.h protobuf interface, next/ prototype)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ontend The core (libtransport) no longer uses XMPP element types: no Swift::JID, Presence, Message, IQ, VCard, RosterPayload, DiscoInfo or MUC payloads. Users are strings (the bare JID for XMPP), buddies and rooms are legacy names, statuses and messages are pbnetwork types. Swift is still used for the event loop, timers and networking (next step: Asio), and by UserRegistry (XMPP server mode, still in libtransport). Swift:: uses in libtransport + include/transport: ~620 -> 75, all of them networking, timers, the event loop and UserRegistry. Core interface -------------- - Frontend: creates User, RosterManager, UserManager and (new) Conversation; handleLoginResult (backend accepted/rejected a login); sendRawXML / onRawXMLReceived for features.rawxml; room list takes legacy room names. Events use pbnetwork::VCard / pbnetwork::Buddy. sendPresence, sendMessage, sendIQ, sendRosterRequest, sendCapabilitiesRequest, onPresenceReceived, onMessageReceived, onCapabilitiesReceived, onAvailableChanged are gone. - User: getUserName(); setReadyToConnect(); onStatusChanged(pbnetwork::Status); joinRoom/leaveRoom (joins are postponed until connected and redone on reconnect) with handleRoomJoined hook; disconnectUser takes pbnetwork::ConnectionError. Presence aggregation, caps and the connecting timeout moved to XMPPUser. - UserManager: loginUser (vip_only, enable_transport, create, onUserCreated) and autoRegisterUser; doRemoveUser hook. Presence-driven login, subscriptions, connect/disconnectUser and the remove timer moved to XMPPUserManager. - Buddy/LocalBuddy: status is pbnetwork::StatusType; JIDs, presence stanzas and raw presences moved to XMPPRosterManager (generatePresenceStanzas, handleRawPresence) and jidutils (getBuddyJID, getSafeName, JIDToLegacyName, JIDToBuddy, buddyFlagsFromJID). RosterManager gets doSendBuddyStatus, doRequestAuthorization, isValidBuddyName hooks. - Conversation: messages are (WrapperMessage_Type, ConversationMessage): TYPE_CONV_MESSAGE, TYPE_ROOM_SUBJECT_CHANGED, TYPE_ATTENTION, TYPE_CONV_MESSAGE_ACK, TYPE_BUDDY_TYPING/TYPED/STOPPED_TYPING. The core keeps the participant list (pbnetwork::Participant); handleMessage and doHandleParticipantChanged are implemented by the frontend's subclass. The backend listens on onMessageToSend (NetworkConversation is gone). - ConversationManager: getConversationForMessage / handleMessageReceived (PM detection via ConversationMessage.pm + nickname). - AdminInterface::handleQuery(string) returns the reply; the admin_jid check and the reply stanza moved to XMPPFrontend. - NetworkPluginServer: no XMPP parser/serializer; chat states, receipts, attentions, subjects, xhtml/OOB media wrapping and timestamps are rendered by XMPPConversation. XMPP frontend ------------- - XMPPConversation: everything XMPP from the old Conversation (resources, MUC presences, subject ordering, caching, carbons, XEP-0356) plus the media wrapping from NetworkPluginServer. - PresenceOracle moved here; XMPPFrontend feeds it and routes presences to XMPPUserManager as Component did before. Raw XML parsing/serializing (features.rawxml) moved here from NetworkPluginServer. Slack frontend -------------- - No more simulated XMPP presences/messages: SlackSession logs the user in via UserManager::loginUser, joins/leaves rooms via User::joinRoom / leaveRoom and sends Slack messages through ConversationManager::handleMessageReceived. SlackConversation posts TYPE_CONV_MESSAGE to Slack. Channel maps are keyed by legacy room name. Behaviour changes ----------------- - onReadyToConnect fires once: before, a client with known caps could trigger it twice (two Login messages to the backend). - Raw-XML stanzas sent to the backend are copies; before, the user's presence held by the PresenceOracle got its "to" rewritten. - Slack connects the legacy network at login instead of after the 5 s caps timeout, and posts only text messages (no empty chat-state posts). Tests ----- Unit tests ported: TestingFactory creates conversations through the frontend; assertions on messages to the backend check pbnetwork types. Not verified ------------ Not compiled or run here (no local builds); CI is the check. Linux-only code reviewed by reading only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Config.h no longer pulls them in through Boost headers; the Linux builds failed.
…s pbnetwork::StatusType
vitalyster
force-pushed
the
core-callbacks-wip
branch
from
October 24, 2026 11:45
ac3014f to
65e82cc
Compare
…ine parser transport-protocol no longer uses Boost. Config keeps its interface: - ConfigValue (std::variant of string, int, bool, list, vector) replaces boost::program_options::variable_value; as<T>() throws on a type mismatch as before, so CONFIG_*_DEFAULTED and safeAs keep their meaning. - inih (r62, BSD, vendored in protocol/inih) parses the config files. '#' starts a comment anywhere on a line, as with program_options; multi-line values are off and lines can be of any length. - The registered variables are a table with the same keys, types and defaults. Storing follows program_options: the first value given is final (command line before the file, earlier loads before reload), defaults are replaced, a registered variable set twice in one file is an error, multi-value variables accumulate. - CommandLine replaces program_options in Config::createFromArgs and in spectrum2's main(); same options, same "-x value", "--name=value" and positional config file forms, unknown options ignored. Changes: syntax errors and invalid int/bool values make load() print the error and return false instead of throwing out of it. transport-protocol stops linking Boost; libtransport links it itself. Files that got Boost headers through Config.h include them directly. New tests cover comments, types, defaults, duplicates, $jid, reload and CommandLine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ON by default. ENABLE_SLACK_FRONTEND is now a dependent option: it can only be turned on together with ENABLE_XMPP_FRONTEND and is OFF otherwise. With ENABLE_XMPP_FRONTEND OFF neither frontend is built, spectrum2 doesn't link them, and libtransport_test, which compiles the frontend sources in, isn't built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IRC backend reads Login.legacyName as "nick@server[:port]", as the libpurple IRC backend does; a bare nick still uses the server given on the command line. Errors name the server it could not reach. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- client_json knows TYPE_JOIN_ROOM / TYPE_LEAVE_ROOM (pbnetwork::Room), so the frontend relays them to the backend. - The IRC backend sends JOIN on JoinRoom (messages sent before the join is confirmed are queued, no second JOIN) and PART on LeaveRoom. A non-empty Login.password is sent as PASS before NICK (on Libera it identifies the nick to NickServ); PASS lines are not logged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The IRC backend tracks who is in each channel and sends TYPE_PARTICIPANT_CHANGED (pbnetwork::Participant) for the NAMES list (~ & @ % prefixes are PARTICIPANT_FLAG_MODERATOR, our own nick PARTICIPANT_FLAG_ME), JOIN, PART, KICK (FLAG_KICKED), QUIT (gone from every channel the nick was in) and NICK (newname); gone is STATUS_NONE as elsewhere in Spectrum. The frontend relays them to the user and client_json knows the Participant payload. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /api/login, /api/send, /api/logout and a chunked text/event-stream at GET /api/events that resumes from Last-Event-ID. Meant to sit behind nginx for HTTPS; listens on 127.0.0.1 by default. A session with no open event stream for 90 s is logged out at the backend.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First steps toward building the spectrum2 core (transport-protocol, libtransport, spectrum2, transport-plugin) without Swiften and Boost, so that spectrum2 can serve non-XMPP clients through a raw protobuf frontend. The XMPP and Slack frontends keep Swiften/Boost.
This PR covers three steps:
Util::toLowerUTF8) and base64.Event<Sig>callbacks instead of boost::signals2. Each event has a single listener; where a second listener existed, it chains to the first.pbnetworktypes.It also adds
next/, a standalone prototype of a protobuf frontend plus an IRC backend. It is off by default (ENABLE_NEXT).Design
Frontendcreates theUser,RosterManager,UserManagerandConversationsubclasses. It reports login results and carries raw XML in both directions (features.rawxml). The XMPP-specific send* and on* members are gone.(WrapperMessage_Type, ConversationMessage)pairs: plain message, subject, attention, delivery receipt, and the three typing states.NetworkConversationis gone; the backend listens onConversation::onMessageToSend.RosterManager:doSendBuddyStatus,doRequestAuthorization,isValidBuddyNameUser:handleRoomJoined,sendCurrentPresenceUserManager:doRemoveUserUser:setReadyToConnect,onStatusChanged(pbnetwork::Status), andjoinRoom/leaveRoom. Joins wait until the user is connected and are redone after a reconnect.UserManager:loginUser(checks vip_only and enable_transport) andautoRegisterUser.XMPPUserXMPPUserManagerXMPPConversationPresenceOracle, raw-XML parsing, and JID↔legacy-name mapping (newjidutils)loginUser,joinRoom/leaveRoomandConversationManager::handleMessageReceiveddirectly.AdminInterface::handleQuery(string)returns the reply. The admin JID check moved intoXMPPFrontend.Behaviour changes
Not done yet
UserRegistry(XMPP server mode) still lives in libtransport.Testing
next/: not compiled locally; CI is the first check.