Side-loading apps, which bypasses the Windows Store, is typically used for enterprise apps as well as running private betas and handling other limited-distribution scenarios. In many cases, you probably do want to collect feedback from your users. However, side-loaded apps do not have the Rate and Review command in their settings panels as do those that are acquired from the Store (see Guidelines for app settings). So the easy answer to the question in this post’s title is no, at least not automatically through the Store mechanism.
Is there a backdoor, perhaps? Nope. Side-loading means there isn’t a record in the Store database for the app, so there’s no app identity to which to attach ratings and reviews.
That said, the goal of ratings and review for a side-loaded app is very different than for those in the Store. In the latter case, ratings and reviews are important for marketing and visibility of the app. For more limited distribution, that’s not the case, and you’re really looking primarily to collect actionable feedback.
With that in mind, there are a few incremental approaches you can take depending on how much data you want. For starters, just a simple mailto: link in a settings command will make it easy to email feedback. This takes only a couple lines of code (see Mixing links and flyouts in Settings (WinJS) for some examples of links).
With a little more work, you can set up a web page with a feedback form, show that in a settings flyout using an iframe or webview, and have that page email the results to you. Another incremental step would be to have the page save feedback in a simple database, so you could query it and save yourself the data-entry step from the email approach.
In fact, for a beta app, having queryable data like this is much more useful than the data the Store could collect, which you can only browse on your product page anyway. A web form, as described above, could also gather more specific feedback that you wouldn’t get through a simple Review text field.
With all this in mind, putting in the extra work to collect feedback is well worth it!