Depends:
If you need Unity engine functionality (physics, pathfinding, etc) then I think it'd be easier for you to implement the few modifications it'd take for a headless client.
If you don't need Unity engine functionality, and I'm guessing maybe you don't given your note about rolling your own A* pathfinding - which I assume means you can handle collision detection & line of sight on your own too - then I think it'd be easier for you to implement some modifications to the server.
However, both approaches do require modifications to TNet itself. This means digging through TNet's code, understanding it, and rewriting it to work how you want it to. If that's what's offputting about Forge then I'm not sure if the switch would be a good idea for you. TNet is written really well and like I said earlier you'll still find value even if you strip it to its core, but work is work and you might not have the time or resources for these modifications.
To address your concerns more specifically, though, TNet works mostly through RFCs (aka RPCs, literally the same thing) by default. You decorate a function with the [RFC] attribute then use TNObject.Send(...) to call that function. TNet then takes the player ID (TCP connection UID, basically), channel ID, UID of the TNObject being used, the function name (or ID if an ID is used in the decoration), and the parameters you pass to it, then wraps this all up in a Buffer (TNet custom memory class w/ built in pooling) and sets a packet identifier (using a Packet enum, eg; Packet.ForwardToAll) before sending it off to the server. The server receives the Buffer, reads the packet identifier, and handles it accordingly. There exists functionality within TNet to use custom packet identifiers and custom packet handlers on both the client and server, so you can see how it might be easy for you to implement your server-authoritative logic. Additionally, there's functionality to write and send packets directly if you don't want TNet to wrap it up like an RFC. Under the hood, there's extraordinary serialization support for most (maybe all!?) data types, including Unity objects (literally entire GameObjects, including the Components!). Though, as a sufferer of micro-optimization-itus, I must recommend against sending an entire GameObject over the network.
Also, multiple channels can exist on a single server. Channels can be used to essentially network-LOD a very large world, or can be used like separate "rooms". So you might be able to cut down on server costs by packing as many channels (and players) into a server.
If you do decide to go with TNet, here's some tips on where to get started with understanding and rewriting it:
The Player and TcpProtocol classes mostly handle the low-level connection & packet handling. I believe the first step in the "game" connection process occurs in TcpProtocol, but other than that you shouldn't have to touch anything here.
TNManager is essentially a wrapper around GameClient, with some extra functionality sprinkled in.
GameClient handles the actual packet processing on the client. All packets are first processed here, even custom packets.
TNObject handles client-side RFC processing, after being pre-processed by GameClient. An RFC will execute on the same GameObject you send it from on each Client. Check TNObject.isMine if you need to determine if that GameObject is local or remote.
Channel class probably won't need to be modified - understand the concept (and its applications) before trying to understand the code.
GameServer class is meaty. All packets are processed here, even custom packets. Also contains connection & channel management and saving / loading. Take note of the delegates made available and hook up your listeners.
DataNode is a tree-based class with full local and network serialization support.
Buffer is kind of a pooled MemoryStream class, you can use it for stuff other than packet construction.
Serialization (not accessed directly, just a *ton* of extension methods) is meaty as hell. If you run into unexplainable errors, it's likely you'll find something here in the callstack. Make note of the IBinarySerializable interface declaration. I use this interface for everything.
