To export your library out of Papers 3, you will want to follow a 2-step process: metadata then PDFs
To get the metadata file - go to File > Export > and select Refman RIS. Save that file somewhere you can easily find it (like your Desktop).
Next, search for library.papers3 and click on it. You'll see a bunch of folders within it including "Articles" Checkpoints" "Styles".... Now search for .pdf in that library.Papers3 folder and those will be all the PDFs from your library.
The new tool you will use will likely advise on how best to import, but generally, you will import the metadata file first and then drag in the PDFs.
Let us know if you have any other questions